doubts on pci bar0 physical address getting changed in Win2k8R2

Hi,

I am working on an hba that exposes two pci functions 0 and 1 (let’s call them pf0 and pf1). When the os is booting up, prior to the invocation of DriverEntry for my storport based miniport driver, I have checked the bar0 physical addresses in WinDBG for these two pci functions using !pci. Say bar0 physical address for pf0 is ‘abcd0000’ and pf1 is ‘abef0000’.

After the DriverEntry completes, if HwStorFindAdapter gets invoked for pf1 first instead of pf0, I am seeing that the bar0 address of pf1 is getting changed to ‘abcd0000’ which is same as that of pf0. I mean, both the pci functions at this point have the same bar0 physical address. This change is happening even before the miniport driver is getting invoked at HwStorFindAdapter but after returning from DriverEntry. Though StorPortGetDeviceBase is returning a virtual address for this changed bar0 map for pf1, the pci config space, if read from the miniport driver, is giving completely invalid values (read 0xff; I know for sure some of the hba registers can’t have this value based on the hardware spec).

But if the HwStorFindAdapter gets called for pf0 first, the bar0 addresses for both the pci functions are retained as assigned by the system bios and are distinct. And when I do a pci read to the hba registers, I am getting valid values.

BTW, the server is IBM x3650 M1 box having 1G ram and is running windows server 2008 R2. We are seeing this issue only on four of our lab servers having exactly this same configuration.

Here are my questions:

  1. Under what all situations can the physical address for bar0 change in the context of an os?
  2. If it changes due to a pci rebalancing event, why is it that bar0 for pf1 is getting the same value as that of pf0?
  3. In continuation, shouldn’t pci driver be checking if the bar0 address that it is going to assign to one pci function was already assigned to some other pci function by the system bios?
  4. Say, both the pci functions are disabled through device manager. Now, if I do !dd on the bar0 physical address + an offset for one of my hba registers, the output says "Physical memory read at
failed". If this is the case, in the problem context mentioned above, shouldn't reads across pci bar0 address for pf1 after the rebalancing event should give valid results (ignoring the bar0 address of pf0, whatever it might be)? Why am I getting all invalid values for my registers?

My main point is to understand how this whole thing behaves. Thanks in advance.

- kk.

PCI interface in your ASIC has a design bug in its power management and configuration support.

There seems to be interference between PCI functions and wrong assumptions about function 0 being initialized first.

Before a device is started, the PCI function may be put in D3. If your ASIC cannot support that, tough luck.

Hi Alex,

The hardware spec says that the asic supports the D3 state. Also on other servers with the same hba (read a different hba based out of same asic and same revision), I had seen pci function 1 coming up first without any issues. In fact, countless times had I seen this hba coming up without any problems. And this is the first time we are seeing this issue in the past 4+ years. So I would rule out any assumptions in the asic about pci function 0 being initialized first.

  • kk.

If you enable F1 first, and then enable F0, does the device function?

When a devnode is disabled, the PCI function may be put in D3 state by PCI.SYS, and its configuration state is invalid. You should not access BARs of a device which is in disabled state. Other PCI devices can reclaim its mapping.

To debug this problem, you may need to write a small non-storport driver which will allow to change the device D-states and probe the config space.

While in this failure state, dump the configuration space of both functions (!pci) and post here.

I’ve seen bar changing between preboot and OS start mostly on oo-e-fee
(UEFI) systems.
Hook up a bus analyzer, you should see the config cycle that your bar0 on
PF1 is being up updated. Now you want to check if you see config cycle for
PF0 bar 0. If yes, that means the new bar address somehow doesn’t stick so
you can’t blame msft.

If you don’t see PF0 bar0 but only PF1 bar0 update and PF1 bar0 was updated
to that of PF0’s bar0, I would focus on platform specific issue.

Good luck,
Calvin

On Fri, Oct 18, 2013 at 2:51 AM, wrote:

> Hi,
>
> I am working on an hba that exposes two pci functions 0 and 1 (let’s call
> them pf0 and pf1). When the os is booting up, prior to the invocation of
> DriverEntry for my storport based miniport driver, I have checked the bar0
> physical addresses in WinDBG for these two pci functions using !pci. Say
> bar0 physical address for pf0 is ‘abcd0000’ and pf1 is ‘abef0000’.
>
> After the DriverEntry completes, if HwStorFindAdapter gets invoked for pf1
> first instead of pf0, I am seeing that the bar0 address of pf1 is getting
> changed to ‘abcd0000’ which is same as that of pf0. I mean, both the pci
> functions at this point have the same bar0 physical address. This change is
> happening even before the miniport driver is getting invoked at
> HwStorFindAdapter but after returning from DriverEntry. Though
> StorPortGetDeviceBase is returning a virtual address for this changed bar0
> map for pf1, the pci config space, if read from the miniport driver, is
> giving completely invalid values (read 0xff; I know for sure some of the
> hba registers can’t have this value based on the hardware spec).
>
> But if the HwStorFindAdapter gets called for pf0 first, the bar0 addresses
> for both the pci functions are retained as assigned by the system bios and
> are distinct. And when I do a pci read to the hba registers, I am getting
> valid values.
>
> BTW, the server is IBM x3650 M1 box having 1G ram and is running windows
> server 2008 R2. We are seeing this issue only on four of our lab servers
> having exactly this same configuration.
>
> Here are my questions:
> 1. Under what all situations can the physical address for bar0 change in
> the context of an os?
> 2. If it changes due to a pci rebalancing event, why is it that bar0 for
> pf1 is getting the same value as that of pf0?
> 3. In continuation, shouldn’t pci driver be checking if the bar0 address
> that it is going to assign to one pci function was already assigned to some
> other pci function by the system bios?
> 4. Say, both the pci functions are disabled through device manager. Now,
> if I do !dd on the bar0 physical address + an offset for one of my hba
> registers, the output says “Physical memory read at failed”. If
> this is the case, in the problem context mentioned above, shouldn’t reads
> across pci bar0 address for pf1 after the rebalancing event should give
> valid results (ignoring the bar0 address of pf0, whatever it might be)? Why
> am I getting all invalid values for my registers?
>
> My main point is to understand how this whole thing behaves. Thanks in
> advance.
>
> - kk.
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Thanks Alex and Calvin for your replies.

When I enable pf1 first through device manager, I am running into the same issue that I had described. So the pf1 doesn’t get initialized.

This answers my questions 2 & 3. Can you also please answer my questions 1 & 4?

0: kd> !pci 101 24 0 0

PCI Configuration Space (Segment:0000 Bus:24 Device:00 Function:00)
Common Header:
00: VendorID 1657 Brocade Communications Systems
02: DeviceID 0013
04: Command 0400 InterruptDis
06: Status 0010 CapList
08: RevisionID 01
09: ProgIF 00
0a: SubClass 04 Fibre Channel
0b: BaseClass 0c Serial Bus Controller
0c: CacheLineSize 0010 BurstDisabled
0d: LatencyTimer 00
0e: HeaderType 80
0f: BIST 00
10: BAR0 c5fe0004
14: BAR1 00000000
18: BAR2 c5fdc004
1c: BAR3 00000000
20: BAR4 00000000
24: BAR5 00000000
28: CBCISPtr 00000000
2c: SubSysVenID 1657
2e: SubSysID 0014
30: ROMBAR 00000000
34: CapPtr 40
3c: IntLine 0a
3d: IntPin 01
3e: MinGnt 00
3f: MaxLat 00
Device Private:
40: 00035001 1400200b 00000000 00000000
50: 00176011 00000002 00001002 00000000
60: 3c020010 10008c84 000b5816 00032482
70: 10410000 00000000 00000000 00000000
80: 00000000 00000013 00000000 00000000
90: 00000002 00000000 00000000 00000000
a0: 00000000 00000000 00000000 00000000
b0: 00000000 00000000 00000000 00000000
c0: 00000000 00000000 00000000 00000000
d0: 00000000 00000000 00000000 00000000
e0: 00000000 00000000 00000000 00000000
f0: 00000000 00000000 00000000 00000000
Capabilities:
40: CapID 01 PwrMgmt Capability
41: NextPtr 50
42: PwrMgmtCap 0003 Version=3
44: PwrMgmtCtrl 200b DataScale:1 DataSel:0 D3Hot

50: CapID 11 MSI-X Capability
51: NextPtr 60
52: MsgCtrl TableSize:0x017 FuncMask:0 MSIXEnable:0
54: MSIXTable 00000002 ( BIR:2 Offset:0x0 )
58: PBATable 00001002 ( BIR:2 Offset:0x1000 )

60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 3c02 Type:Endpoint
64: Device Caps 10008c84
68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce
6a: Device Status 000b tp ap UR fe NF CE
6c: Link Caps 00032482
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5

84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp
74: Slot Caps 00000000
78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab
7a: Slot Status 0000 pds hpi cc pdc ms pfd ab
7c: Root Control 0000 pmei fs nfs cs
7e: Reserved 0000
80: Root Status 00000000 pmep pmes ID:0

Enhanced Capabilities:
100: CapID 0001 Advanced Error Reporting Capability
Version 1
NextPtr 140

140: CapID 0002 Virtual Channel Capability
Version 1
NextPtr 180

180: CapID 0004 Power Budgeting Capability
Version 1
NextPtr 000

0: kd> !pci 101 24 0 1

PCI Configuration Space (Segment:0000 Bus:24 Device:00 Function:01)
Common Header:
00: VendorID 1657 Brocade Communications Systems
02: DeviceID 0013
04: Command 0406 MemSpaceEn BusInitiate InterruptDis
06: Status 0010 CapList
08: RevisionID 01
09: ProgIF 00
0a: SubClass 04 Fibre Channel
0b: BaseClass 0c Serial Bus Controller
0c: CacheLineSize 0010 BurstDisabled
0d: LatencyTimer 00
0e: HeaderType 80
0f: BIST 00
10: BAR0 c5fe0004
14: BAR1 00000000
18: BAR2 c5fdc004
1c: BAR3 00000000
20: BAR4 00000000
24: BAR5 00000000
28: CBCISPtr 00000000
2c: SubSysVenID 1657
2e: SubSysID 0014
30: ROMBAR 00000000
34: CapPtr 40
3c: IntLine 00
3d: IntPin 02
3e: MinGnt 00
3f: MaxLat 00
Device Private:
40: 00035001 14002008 00000000 00000000
50: 80176011 00002002 00003002 00000000
60: 3e020010 10008c84 00095816 00032482
70: 10410000 00000000 00000000 00000000
80: 00000000 00000013 00000000 00000000
90: 00000000 00000000 00000000 00000000
a0: 00000000 00000000 00000000 00000000
b0: 00000000 00000000 00000000 00000000
c0: 00000000 00000000 00000000 00000000
d0: 00000000 00000000 00000000 00000000
e0: 00000000 00000000 00000000 00000000
f0: 00000000 00000000 00000000 00000000
Capabilities:
40: CapID 01 PwrMgmt Capability
41: NextPtr 50
42: PwrMgmtCap 0003 Version=3
44: PwrMgmtCtrl 2008 DataScale:1 DataSel:0 D0

50: CapID 11 MSI-X Capability
51: NextPtr 60
52: MsgCtrl TableSize:0x017 FuncMask:0 MSIXEnable:1
54: MSIXTable 00002002 ( BIR:2 Offset:0x2000 )
58: PBATable 00003002 ( BIR:2 Offset:0x3000 )

60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 3e02 Type:Endpoint
64: Device Caps 10008c84
68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce
6a: Device Status 0009 tp ap UR fe nf CE
6c: Link Caps 00032482
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5

84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp
74: Slot Caps 00000000
78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab
7a: Slot Status 0000 pds hpi cc pdc ms pfd ab
7c: Root Control 0000 pmei fs nfs cs
7e: Reserved 0000
80: Root Status 00000000 pmep pmes ID:0

Enhanced Capabilities:
100: CapID 0001 Advanced Error Reporting Capability
Version 1
NextPtr 180

180: CapID 0004 Power Budgeting Capability
Version 1
NextPtr 000

Also I have one more question. In the error scenario that I have mentioned, will the root complex forward the mem read on “bar0 + offset” to pf0 and pf0 rejects it or will it forward to only PF1 (as it is in enabled state)?

  • kk.

In this instance having the BARs for both functions the same should not
cause a problem since Memory Space Enable in the PCIe Command register is
only set for one of the functions. Later, when Function 0 is initialized,
its BAAR will be changed before that function’s Memory Space Enable bit is
set.

xxxxx@gmail.com
Sent by: xxxxx@lists.osr.com
10/21/2013 03:37 AM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
RE:[ntdev] doubts on pci bar0 physical address getting changed in Win2k8R2

Thanks Alex and Calvin for your replies.



When I enable pf1 first through device manager, I am running into the same
issue that I had described. So the pf1 doesn’t get initialized.



This answers my questions 2 & 3. Can you also please answer my questions 1
& 4?



0: kd> !pci 101 24 0 0

PCI Configuration Space (Segment:0000 Bus:24 Device:00 Function:00)
Common Header:
00: VendorID 1657 Brocade Communications Systems
02: DeviceID 0013
04: Command 0400 InterruptDis
06: Status 0010 CapList
08: RevisionID 01
09: ProgIF 00
0a: SubClass 04 Fibre Channel
0b: BaseClass 0c Serial Bus Controller
0c: CacheLineSize 0010 BurstDisabled
0d: LatencyTimer 00
0e: HeaderType 80
0f: BIST 00
10: BAR0 c5fe0004
14: BAR1 00000000
18: BAR2 c5fdc004
1c: BAR3 00000000
20: BAR4 00000000
24: BAR5 00000000
28: CBCISPtr 00000000
2c: SubSysVenID 1657
2e: SubSysID 0014
30: ROMBAR 00000000
34: CapPtr 40
3c: IntLine 0a
3d: IntPin 01
3e: MinGnt 00
3f: MaxLat 00
Device Private:
40: 00035001 1400200b 00000000 00000000
50: 00176011 00000002 00001002 00000000
60: 3c020010 10008c84 000b5816 00032482
70: 10410000 00000000 00000000 00000000
80: 00000000 00000013 00000000 00000000
90: 00000002 00000000 00000000 00000000
a0: 00000000 00000000 00000000 00000000
b0: 00000000 00000000 00000000 00000000
c0: 00000000 00000000 00000000 00000000
d0: 00000000 00000000 00000000 00000000
e0: 00000000 00000000 00000000 00000000
f0: 00000000 00000000 00000000 00000000
Capabilities:
40: CapID 01 PwrMgmt Capability
41: NextPtr 50
42: PwrMgmtCap 0003 Version=3
44: PwrMgmtCtrl 200b DataScale:1 DataSel:0 D3Hot

50: CapID 11 MSI-X Capability
51: NextPtr 60
52: MsgCtrl TableSize:0x017 FuncMask:0 MSIXEnable:0
54: MSIXTable 00000002 ( BIR:2 Offset:0x0 )
58: PBATable 00001002 ( BIR:2 Offset:0x1000 )

60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 3c02 Type:Endpoint
64: Device Caps 10008c84
68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce
6a: Device Status 000b tp ap UR fe NF CE
6c: Link Caps 00032482
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5

84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128
noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom
ltr OBFF:0 eetlp
74: Slot Caps 00000000
78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab
7a: Slot Status 0000 pds hpi cc pdc ms pfd ab
7c: Root Control 0000 pmei fs nfs cs
7e: Reserved 0000
80: Root Status 00000000 pmep pmes ID:0

Enhanced Capabilities:
100: CapID 0001 Advanced Error Reporting Capability
Version 1
NextPtr 140

140: CapID 0002 Virtual Channel Capability
Version 1
NextPtr 180

180: CapID 0004 Power Budgeting Capability
Version 1
NextPtr 000

0: kd> !pci 101 24 0 1

PCI Configuration Space (Segment:0000 Bus:24 Device:00 Function:01)
Common Header:
00: VendorID 1657 Brocade Communications Systems
02: DeviceID 0013
04: Command 0406 MemSpaceEn BusInitiate InterruptDis
06: Status 0010 CapList
08: RevisionID 01
09: ProgIF 00
0a: SubClass 04 Fibre Channel
0b: BaseClass 0c Serial Bus Controller
0c: CacheLineSize 0010 BurstDisabled
0d: LatencyTimer 00
0e: HeaderType 80
0f: BIST 00
10: BAR0 c5fe0004
14: BAR1 00000000
18: BAR2 c5fdc004
1c: BAR3 00000000
20: BAR4 00000000
24: BAR5 00000000
28: CBCISPtr 00000000
2c: SubSysVenID 1657
2e: SubSysID 0014
30: ROMBAR 00000000
34: CapPtr 40
3c: IntLine 00
3d: IntPin 02
3e: MinGnt 00
3f: MaxLat 00
Device Private:
40: 00035001 14002008 00000000 00000000
50: 80176011 00002002 00003002 00000000
60: 3e020010 10008c84 00095816 00032482
70: 10410000 00000000 00000000 00000000
80: 00000000 00000013 00000000 00000000
90: 00000000 00000000 00000000 00000000
a0: 00000000 00000000 00000000 00000000
b0: 00000000 00000000 00000000 00000000
c0: 00000000 00000000 00000000 00000000
d0: 00000000 00000000 00000000 00000000
e0: 00000000 00000000 00000000 00000000
f0: 00000000 00000000 00000000 00000000
Capabilities:
40: CapID 01 PwrMgmt Capability
41: NextPtr 50
42: PwrMgmtCap 0003 Version=3
44: PwrMgmtCtrl 2008 DataScale:1 DataSel:0 D0

50: CapID 11 MSI-X Capability
51: NextPtr 60
52: MsgCtrl TableSize:0x017 FuncMask:0 MSIXEnable:1
54: MSIXTable 00002002 ( BIR:2 Offset:0x2000 )
58: PBATable 00003002 ( BIR:2 Offset:0x3000 )

60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 3e02 Type:Endpoint
64: Device Caps 10008c84
68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce
6a: Device Status 0009 tp ap UR fe nf CE
6c: Link Caps 00032482
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5

84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128
noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom
ltr OBFF:0 eetlp
74: Slot Caps 00000000
78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab
7a: Slot Status 0000 pds hpi cc pdc ms pfd ab
7c: Root Control 0000 pmei fs nfs cs
7e: Reserved 0000
80: Root Status 00000000 pmep pmes ID:0

Enhanced Capabilities:
100: CapID 0001 Advanced Error Reporting Capability
Version 1
NextPtr 180

180: CapID 0004 Power Budgeting Capability
Version 1
NextPtr 000

Also I have one more question. In the error scenario that I have
mentioned, will the root complex forward the mem read on “bar0 + offset”
to pf0 and pf0 rejects it or will it forward to only PF1 (as it is in
enabled state)?

- kk.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

There is nothing wrong in the configuration. The function 1 is in D0, and its BARs are enabled. The function 0 is in D3hot, and its BARs are disabled. The interrupt disable bit is set for F1 (does it affect the legacy IRQ only?).

If you do a !dd in this state, can you read the BAR of PF1?

  1. The OS can change addresses allocated by BIOS, and there is nothing wrong with that.
  2. Until the device is started, its BARs may be disabled by PCI.SYS, and be non-readable.

This is because I had broken into debugger at HwStorInitialize routine and the interrupts were not enabled at this point.

!dd c5fe0000+14108
#c5ff4108 ffffffff ffffffff ffffffff ffffffff
#c5ff4118 ffffffff ffffffff ffffffff ffffffff
#c5ff4128 ffffffff ffffffff ffffffff ffffffff
#c5ff4138 ffffffff ffffffff ffffffff ffffffff
#c5ff4148 ffffffff ffffffff ffffffff ffffffff
#c5ff4158 ffffffff ffffffff ffffffff ffffffff
#c5ff4168 ffffffff ffffffff ffffffff ffffffff
#c5ff4178 ffffffff ffffffff ffffffff ffffffff

As per the hw spec, the value at offset 14108 can’t be 0xff. The reset value after pci bus reset is specified as zero. I have tried !dd with other registers as well and the result is same.

  • kk.

>In the error scenario that I have mentioned, will the root complex forward the mem read on “bar0 + offset” to pf0 and pf0 rejects it or will it forward to only PF1 (as it is in enabled state)?

The root complex doesn’t care if it goes to PF0 or PF1, because it’s the same route. It’s up to the device to decode the function and decide whether to respond or not.

So far it looks like the ASIC has a bug. It’s time to get a PCIe analyzer and see what’s going on.

Try the following. Using the !ecd (edit configuration, DWORD) command, disable the memory access of PF1, set PF0 to D0, enable the memory access of PF0, see if you can read the BAR.

Return all back, check that the BAR is not readable. Now set PF0 BAR0 to 45fe0004, see if the physical address c5fe0000+x is now readable.