I am working on an hba that exposes two pci functions 0 and 1 (let’s call them pf0 and pf1). When the os is booting up, prior to the invocation of DriverEntry for my storport based miniport driver, I have checked the bar0 physical addresses in WinDBG for these two pci functions using !pci. Say bar0 physical address for pf0 is ‘abcd0000’ and pf1 is ‘abef0000’.
After the DriverEntry completes, if HwStorFindAdapter gets invoked for pf1 first instead of pf0, I am seeing that the bar0 address of pf1 is getting changed to ‘abcd0000’ which is same as that of pf0. I mean, both the pci functions at this point have the same bar0 physical address. This change is happening even before the miniport driver is getting invoked at HwStorFindAdapter but after returning from DriverEntry. Though StorPortGetDeviceBase is returning a virtual address for this changed bar0 map for pf1, the pci config space, if read from the miniport driver, is giving completely invalid values (read 0xff; I know for sure some of the hba registers can’t have this value based on the hardware spec).
But if the HwStorFindAdapter gets called for pf0 first, the bar0 addresses for both the pci functions are retained as assigned by the system bios and are distinct. And when I do a pci read to the hba registers, I am getting valid values.
BTW, the server is IBM x3650 M1 box having 1G ram and is running windows server 2008 R2. We are seeing this issue only on four of our lab servers having exactly this same configuration.
Here are my questions:
Under what all situations can the physical address for bar0 change in the context of an os?
If it changes due to a pci rebalancing event, why is it that bar0 for pf1 is getting the same value as that of pf0?
In continuation, shouldn’t pci driver be checking if the bar0 address that it is going to assign to one pci function was already assigned to some other pci function by the system bios?
Say, both the pci functions are disabled through device manager. Now, if I do !dd on the bar0 physical address + an offset for one of my hba registers, the output says "Physical memory read at
failed". If this is the case, in the problem context mentioned above, shouldn't reads across pci bar0 address for pf1 after the rebalancing event should give valid results (ignoring the bar0 address of pf0, whatever it might be)? Why am I getting all invalid values for my registers?
My main point is to understand how this whole thing behaves. Thanks in advance.
The hardware spec says that the asic supports the D3 state. Also on other servers with the same hba (read a different hba based out of same asic and same revision), I had seen pci function 1 coming up first without any issues. In fact, countless times had I seen this hba coming up without any problems. And this is the first time we are seeing this issue in the past 4+ years. So I would rule out any assumptions in the asic about pci function 0 being initialized first.
If you enable F1 first, and then enable F0, does the device function?
When a devnode is disabled, the PCI function may be put in D3 state by PCI.SYS, and its configuration state is invalid. You should not access BARs of a device which is in disabled state. Other PCI devices can reclaim its mapping.
To debug this problem, you may need to write a small non-storport driver which will allow to change the device D-states and probe the config space.
While in this failure state, dump the configuration space of both functions (!pci) and post here.
I’ve seen bar changing between preboot and OS start mostly on oo-e-fee
(UEFI) systems.
Hook up a bus analyzer, you should see the config cycle that your bar0 on
PF1 is being up updated. Now you want to check if you see config cycle for
PF0 bar 0. If yes, that means the new bar address somehow doesn’t stick so
you can’t blame msft.
If you don’t see PF0 bar0 but only PF1 bar0 update and PF1 bar0 was updated
to that of PF0’s bar0, I would focus on platform specific issue.
Good luck,
Calvin
On Fri, Oct 18, 2013 at 2:51 AM, wrote:
> Hi, > > I am working on an hba that exposes two pci functions 0 and 1 (let’s call > them pf0 and pf1). When the os is booting up, prior to the invocation of > DriverEntry for my storport based miniport driver, I have checked the bar0 > physical addresses in WinDBG for these two pci functions using !pci. Say > bar0 physical address for pf0 is ‘abcd0000’ and pf1 is ‘abef0000’. > > After the DriverEntry completes, if HwStorFindAdapter gets invoked for pf1 > first instead of pf0, I am seeing that the bar0 address of pf1 is getting > changed to ‘abcd0000’ which is same as that of pf0. I mean, both the pci > functions at this point have the same bar0 physical address. This change is > happening even before the miniport driver is getting invoked at > HwStorFindAdapter but after returning from DriverEntry. Though > StorPortGetDeviceBase is returning a virtual address for this changed bar0 > map for pf1, the pci config space, if read from the miniport driver, is > giving completely invalid values (read 0xff; I know for sure some of the > hba registers can’t have this value based on the hardware spec). > > But if the HwStorFindAdapter gets called for pf0 first, the bar0 addresses > for both the pci functions are retained as assigned by the system bios and > are distinct. And when I do a pci read to the hba registers, I am getting > valid values. > > BTW, the server is IBM x3650 M1 box having 1G ram and is running windows > server 2008 R2. We are seeing this issue only on four of our lab servers > having exactly this same configuration. > > Here are my questions: > 1. Under what all situations can the physical address for bar0 change in > the context of an os? > 2. If it changes due to a pci rebalancing event, why is it that bar0 for > pf1 is getting the same value as that of pf0? > 3. In continuation, shouldn’t pci driver be checking if the bar0 address > that it is going to assign to one pci function was already assigned to some > other pci function by the system bios? > 4. Say, both the pci functions are disabled through device manager. Now, > if I do !dd on the bar0 physical address + an offset for one of my hba > registers, the output says “Physical memory read at failed”. If > this is the case, in the problem context mentioned above, shouldn’t reads > across pci bar0 address for pf1 after the rebalancing event should give > valid results (ignoring the bar0 address of pf0, whatever it might be)? Why > am I getting all invalid values for my registers? > > My main point is to understand how this whole thing behaves. Thanks in > advance. > > - kk. > > — > NTDEV is sponsored by OSR > > Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev > > OSR is HIRING!! See http://www.osr.com/careers > > For our schedule of WDF, WDM, debugging and other seminars visit: > http://www.osr.com/seminars > > To unsubscribe, visit the List Server section of OSR Online at > http://www.osronline.com/page.cfm?name=ListServer >
60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 3c02 Type:Endpoint
64: Device Caps 10008c84
68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce
6a: Device Status 000b tp ap UR fe NF CE
6c: Link Caps 00032482
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5
84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp
74: Slot Caps 00000000
78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab
7a: Slot Status 0000 pds hpi cc pdc ms pfd ab
7c: Root Control 0000 pmei fs nfs cs
7e: Reserved 0000
80: Root Status 00000000 pmep pmes ID:0
60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 3e02 Type:Endpoint
64: Device Caps 10008c84
68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce
6a: Device Status 0009 tp ap UR fe nf CE
6c: Link Caps 00032482
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5
84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp
74: Slot Caps 00000000
78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab
7a: Slot Status 0000 pds hpi cc pdc ms pfd ab
7c: Root Control 0000 pmei fs nfs cs
7e: Reserved 0000
80: Root Status 00000000 pmep pmes ID:0
180: CapID 0004 Power Budgeting Capability
Version 1
NextPtr 000
Also I have one more question. In the error scenario that I have mentioned, will the root complex forward the mem read on “bar0 + offset” to pf0 and pf0 rejects it or will it forward to only PF1 (as it is in enabled state)?
In this instance having the BARs for both functions the same should not
cause a problem since Memory Space Enable in the PCIe Command register is
only set for one of the functions. Later, when Function 0 is initialized,
its BAAR will be changed before that function’s Memory Space Enable bit is
set.
60: CapID 10 PCI Express Capability 61: NextPtr 00 62: Express Caps 3c02 Type:Endpoint 64: Device Caps 10008c84 68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce 6a: Device Status 000b tp ap UR fe NF CE 6c: Link Caps 00032482 70: Link Control 0000 es cc rl ld RCB:64 ASPM:None 72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5
84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0 88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp 74: Slot Caps 00000000 78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab 7a: Slot Status 0000 pds hpi cc pdc ms pfd ab 7c: Root Control 0000 pmei fs nfs cs 7e: Reserved 0000 80: Root Status 00000000 pmep pmes ID:0
60: CapID 10 PCI Express Capability 61: NextPtr 00 62: Express Caps 3e02 Type:Endpoint 64: Device Caps 10008c84 68: Device Control 5816 MRR:4K NS ap pf et MP:128 RO ur FE NF ce 6a: Device Status 0009 tp ap UR fe nf CE 6c: Link Caps 00032482 70: Link Control 0000 es cc rl ld RCB:64 ASPM:None 72: Link Status 1041 SCC lt lte NLW:x4 LS:2.5
84: DeviceCaps2 00000013 CTR:3 CTDIS arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0 88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp 74: Slot Caps 00000000 78: Slot Control 0000 pcc PI:?? AI:?? hpi cc pde mrls pfd ab 7a: Slot Status 0000 pds hpi cc pdc ms pfd ab 7c: Root Control 0000 pmei fs nfs cs 7e: Reserved 0000 80: Root Status 00000000 pmep pmes ID:0
180: CapID 0004 Power Budgeting Capability Version 1 NextPtr 000
Also I have one more question. In the error scenario that I have mentioned, will the root complex forward the mem read on “bar0 + offset” to pf0 and pf0 rejects it or will it forward to only PF1 (as it is in enabled state)?
There is nothing wrong in the configuration. The function 1 is in D0, and its BARs are enabled. The function 0 is in D3hot, and its BARs are disabled. The interrupt disable bit is set for F1 (does it affect the legacy IRQ only?).
If you do a !dd in this state, can you read the BAR of PF1?
The OS can change addresses allocated by BIOS, and there is nothing wrong with that.
Until the device is started, its BARs may be disabled by PCI.SYS, and be non-readable.
As per the hw spec, the value at offset 14108 can’t be 0xff. The reset value after pci bus reset is specified as zero. I have tried !dd with other registers as well and the result is same.
>In the error scenario that I have mentioned, will the root complex forward the mem read on “bar0 + offset” to pf0 and pf0 rejects it or will it forward to only PF1 (as it is in enabled state)?
The root complex doesn’t care if it goes to PF0 or PF1, because it’s the same route. It’s up to the device to decode the function and decide whether to respond or not.
So far it looks like the ASIC has a bug. It’s time to get a PCIe analyzer and see what’s going on.
Try the following. Using the !ecd (edit configuration, DWORD) command, disable the memory access of PF1, set PF0 to D0, enable the memory access of PF0, see if you can read the BAR.
Return all back, check that the BAR is not readable. Now set PF0 BAR0 to 45fe0004, see if the physical address c5fe0000+x is now readable.