Time Travel Tracing?

I had a bit of an introduction to a Microsoft tool called Time Travel
Tracing (TTTracer.exe). It’s a tool that they use in support where the
customer runs a client version which traces the program’s flow (without
change to the program) and then they can play it back at Microsoft in a
WinDbg-like environment. I asked, and the tool is not available outside
Microsoft.

I would think that the tool must be doing more than sampling the
Instruction Pointer every 10ms or whatever (as a profiler does) since
they’d need it to have much higher resolution for it to be useful for
that purpose.

So, does anyone have any inkling how they’re doing it?

But the key question is, are there other tools out there that do
something similar? It seems that it could be useful at some point to
most of us.

SoftICE did something similar in the Win95-and-earlier versions, but
they removed the feature for reasons unknown about 10 years back. It
was very useful at times.

The only information I found on the tool were some bullet items here:

http://cs343-spr0607.stanford.edu/index.php/Writeups:Amitabh_Srivastava

Time Travel Tracing?TTTracer basically uses an instruction emulator to
run the code in a sand box which records stuff along the way, which is
also the reason why you won’t see it for kernel-mode code soon.

Since you asked, and since it’s easier to talk about public works,
there are other similar approaches. Here’s a paper that uses
virtualization to accomplish something very similar. Much of Peter
Chen’s work at University of Michigan is in a similar vein. I met him
when he did a talk at Microsoft.

http://www.eecs.umich.edu/virtual/papers/king05_1.pdf

  • Jake Oshins
    Windows Virtualization Guy

“Taed Wynnell” wrote in message
news:xxxxx@ntdev…
I had a bit of an introduction to a Microsoft tool called Time Travel
Tracing (TTTracer.exe). It’s a tool that they use in support where
the customer runs a client version which traces the program’s flow
(without change to the program) and then they can play it back at
Microsoft in a WinDbg-like environment. I asked, and the tool is not
available outside Microsoft.
I would think that the tool must be doing more than sampling the
Instruction Pointer every 10ms or whatever (as a profiler does) since
they’d need it to have much higher resolution for it to be useful for
that purpose.
So, does anyone have any inkling how they’re doing it?
But the key question is, are there other tools out there that do
something similar? It seems that it could be useful at some point to
most of us.
SoftICE did something similar in the Win95-and-earlier versions, but
they removed the feature for reasons unknown about 10 years back. It
was very useful at times.
The only information I found on the tool were some bullet items here:
http://cs343-spr0607.stanford.edu/index.php/Writeups:Amitabh_Srivastava

> I would think that the tool must be doing more than sampling the Instruction Pointer every

10ms or whatever

Why do you think a tool like that would rely upon polling??? Please read about INT 1 in Intel Manual - if TF flag in EFLAGS is set, it gets raised upon every instruction’s execution. This opens a door to countless options - you can trace execution path, modify program flow, etc,etc,etc.

are there other tools out there that do something similar?

You can write your own one - just write your own INT 1 handler, and the rest is limited only by your imagination( and your knowledge of system’s internals and x86 architecture, of course). As long as you limit yourself to user-mode targets, there should not be a problem. However, doing the same with target threads that run in the kernel mode is more tricky, especially on SMP machine…

Anton Bassov

> traces the program’s flow (without change to the program) and then they
can play it back

Not exactly what you are asking for, but JFYI: Watcom has a debugger with
what is called reverse execution capability so that not only you can step
ahead but do the same backwards, restoring the state.

This is obviously not what you want, you cannot, to my knowledge at least,
save the state at a customer’s site to be restored and reverse-executed in
the lab.

If a reverse execution enabled debugger may be a solution for you, google
for it, I saw commercial products that work with MSC.

However, they are not free, watcom IDE is.

Btw, I was wondering, are there any plans to incorporate this reverse
execution feature either in windbg/kd or devenv or both?

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Taed Wynnell
Sent: Tuesday, April 15, 2008 9:48 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Time Travel Tracing?

I had a bit of an introduction to a Microsoft tool called Time Travel
Tracing (TTTracer.exe). It’s a tool that they use in support where the
customer runs a client version which traces the program’s flow (without
change to the program) and then they can play it back at Microsoft in a
WinDbg-like environment. I asked, and the tool is not available outside
Microsoft.

I would think that the tool must be doing more than sampling the Instruction
Pointer every 10ms or whatever (as a profiler does) since they’d need it to
have much higher resolution for it to be useful for that purpose.

So, does anyone have any inkling how they’re doing it?

But the key question is, are there other tools out there that do something
similar? It seems that it could be useful at some point to most of us.

SoftICE did something similar in the Win95-and-earlier versions, but they
removed the feature for reasons unknown about 10 years back. It was very
useful at times.

The only information I found on the tool were some bullet items here:

http://cs343-spr0607.stanford.edu/index.php/Writeups:Amitabh_Srivastava


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Alex Shvedov wrote:

Btw, I was wondering, are there any plans to incorporate this reverse
execution feature either in windbg/kd or devenv or both?

*chuckle*.

Check WinDbg list posts for April 1st this year.

mh.

As regards instruction level tracing backwards for 32 bit windows, it is
implemented through “run trace” feature in Ollydbg.

This is a free tool. But, this is only in user mode.

Chandra

On Thu, Apr 17, 2008 at 6:38 AM, Martin Harvey
wrote:

> Alex Shvedov wrote:
>
> Btw, I was wondering, are there any plans to incorporate this reverse
> > execution feature either in windbg/kd or devenv or both?
> >
> >
> >
>
> chuckle.
>
> Check WinDbg list posts for April 1st this year.
>
> mh.
>
>
> —
>
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

ollydbg run trace is just a log of all instructions executed it doesnt
restore registers and such as it was when the instruction was executed
so it cant be termed as backward whatever

iirc that kind of logs windbg can generate too
with wt (watch and trace )

On 4/17/08, chandra97 97 wrote:
> As regards instruction level tracing backwards for 32 bit windows, it is
> implemented through “run trace” feature in Ollydbg.
>
> This is a free tool. But, this is only in user mode.
>
> Chandra
>
>
> On Thu, Apr 17, 2008 at 6:38 AM, Martin Harvey
> wrote:
> >
> > Alex Shvedov wrote:
> >
> >
> > > Btw, I was wondering, are there any plans to incorporate this reverse
> execution feature either in windbg/kd or devenv or both?
> > >
> > >
> > >
> >
> > chuckle.
> >
> > Check WinDbg list posts for April 1st this year.
> >
> > mh.
> >
> >
> > —
> >
> >
> >
> > NTDEV is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
> >
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
> List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

Ollydbg allows you to trace backwards one instruction at a time after you
stop the run trace. As you step backwards it shows EIP along with other
registers and flags.

On Thu, Apr 17, 2008 at 1:21 PM, raj_r wrote:

> ollydbg run trace is just a log of all instructions executed it doesnt
> restore registers and such as it was when the instruction was executed
> so it cant be termed as backward whatever
>
> iirc that kind of logs windbg can generate too
> with wt (watch and trace )
>
>
>
> On 4/17/08, chandra97 97 wrote:
> > As regards instruction level tracing backwards for 32 bit windows, it is
> > implemented through “run trace” feature in Ollydbg.
> >
> > This is a free tool. But, this is only in user mode.
> >
> > Chandra
> >
> >
> > On Thu, Apr 17, 2008 at 6:38 AM, Martin Harvey
> > wrote:
> > >
> > > Alex Shvedov wrote:
> > >
> > >
> > > > Btw, I was wondering, are there any plans to incorporate this
> reverse
> > execution feature either in windbg/kd or devenv or both?
> > > >
> > > >
> > > >
> > >
> > > chuckle.
> > >
> > > Check WinDbg list posts for April 1st this year.
> > >
> > > mh.
> > >
> > >
> > > —
> > >
> > >
> > >
> > > NTDEV is sponsored by OSR
> > >
> > > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> > >
> > > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> > >
> >
> > — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging
> and
> > other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the
> > List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

yes it shows eip because ollydbg logged it
but other registers are interpreted on actual state of memory when paused

so they are not actual information

here you can see the help saying it explicitly

then backtrace the execution using keys Plus and Minus. Notice that
these keys browse through the history when run trace buffer is closed.
If you step through the run trace, Registers and Information panes get
grayed to emphasize that registers they display are not actual. Trace
buffer doesn’t save top of stack or contents of memory referenced by
registers. Registers, Information and Stack use actual memory state to
interprete registers from the run trace.

regards

raj

On 4/18/08, chandra97 97 wrote:
> Ollydbg allows you to trace backwards one instruction at a time after you
> stop the run trace. As you step backwards it shows EIP along with other
> registers and flags.
>
>
>
> On Thu, Apr 17, 2008 at 1:21 PM, raj_r wrote:
> > ollydbg run trace is just a log of all instructions executed it doesnt
> > restore registers and such as it was when the instruction was executed
> > so it cant be termed as backward whatever
> >
> > iirc that kind of logs windbg can generate too
> > with wt (watch and trace )
> >
> >
> >
> >
> >
> >
> > On 4/17/08, chandra97 97 wrote:
> > > As regards instruction level tracing backwards for 32 bit windows, it is
> > > implemented through “run trace” feature in Ollydbg.
> > >
> > > This is a free tool. But, this is only in user mode.
> > >
> > > Chandra
> > >
> > >
> > > On Thu, Apr 17, 2008 at 6:38 AM, Martin Harvey
> > > wrote:
> > > >
> > > > Alex Shvedov wrote:
> > > >
> > > >
> > > > > Btw, I was wondering, are there any plans to incorporate this
> reverse
> > > execution feature either in windbg/kd or devenv or both?
> > > > >
> > > > >
> > > > >
> > > >
> > > > chuckle.
> > > >
> > > > Check WinDbg list posts for April 1st this year.
> > > >
> > > > mh.
> > > >
> > > >
> > > > —
> > > >
> > > >
> > > >
> > > > NTDEV is sponsored by OSR
> > > >
> > > > For our schedule of WDF, WDM, debugging and other seminars visit:
> > > http://www.osr.com/seminars
> > > >
> > > > To unsubscribe, visit the List Server section of OSR Online at
> > > http://www.osronline.com/page.cfm?name=ListServer
> > > >
> > >
> > > — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging
> and
> >
> >
> >
> > > other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the
> > > List Server section of OSR Online at
> > > http://www.osronline.com/page.cfm?name=ListServer
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
> >
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the
> List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

raj_r wrote:

yes it shows eip because ollydbg logged it
but other registers are interpreted on actual state of memory when paused

so they are not actual information

This is true, but it’s interesting to note that the OllyDbg technique
does a pretty darned good job for most tasks, and it is MUCH less
resource intensive than an instruction-level simulation.

If you want to step back 100 instructions, it isn’t good enough, but to
step back 5 or 10 instructions, it works pretty well.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

yep that way ollydbg does a pretty great work i love it and employ it
for all my user mode needs (windbg lacks too much in user mode)
especially profiling the executed instruction to selectively exclude
from run trace
hit trace etc are pretty usefull

Profile of whole memory, item 0
Count=3946.
Address=7C90305B
First command=REPE SCAS DWORD PTR ES:[EDI]

Profile of whole memory, item 32
Count=119.
Address=004ABE9F
First command=CALL OLLYDBG.004ABCC0

double clicking on this first command will get you the whole block it profiled

004ABE9F |CALL OLLYDBG.004ABCC0 ; \OLLYDBG.004ABCC0
004ABEA4 |ADD ESP,8
004ABEA7 |TEST ESI,ESI
004ABEA9 |JE SHORT OLLYDBG.004ABEAF

you can exclude this block of commands selectively from run trace

lots of neat features and killer context menus make it very good for
user mode debugging

use windbg only on r0 :slight_smile:

regards

raj

On 4/18/08, Tim Roberts wrote:
> raj_r wrote:
> > yes it shows eip because ollydbg logged it
> > but other registers are interpreted on actual state of memory when paused
> >
> > so they are not actual information
> >
> >
>
> This is true, but it’s interesting to note that the OllyDbg technique does a
> pretty darned good job for most tasks, and it is MUCH less resource
> intensive than an instruction-level simulation.
>
> If you want to step back 100 instructions, it isn’t good enough, but to step
> back 5 or 10 instructions, it works pretty well.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

I was out last week and haven’t read this thread, so someone may have already mentioned this, but if not, you might want to check
out Peter Chen’s/Samuel King’s (U. Michigan) work on this subject involving a VM and ReVirt.

Good luck,

mm

Taed Wynnell wrote:

I had a bit of an introduction to a Microsoft tool called Time Travel
Tracing (TTTracer.exe). It’s a tool that they use in support where the
customer runs a client version which traces the program’s flow (without
change to the program) and then they can play it back at Microsoft in a
WinDbg-like environment. I asked, and the tool is not available outside
Microsoft.

I would think that the tool must be doing more than sampling the
Instruction Pointer every 10ms or whatever (as a profiler does) since
they’d need it to have much higher resolution for it to be useful for
that purpose.

So, does anyone have any inkling how they’re doing it?

But the key question is, are there other tools out there that do
something similar? It seems that it could be useful at some point to
most of us.

SoftICE did something similar in the Win95-and-earlier versions, but
they removed the feature for reasons unknown about 10 years back. It
was very useful at times.

The only information I found on the tool were some bullet items here:

_http://cs343-spr0607.stanford.edu/index.php/Writeups:Amitabh_Srivastava_