Hangfire Dashboard

TimesheetSyncTask.Execute

// Job ID: #610251
using Application.Sync;

var timesheetSyncTask = Activate<TimesheetSyncTask>();
await timesheetSyncTask.Execute();
CurrentCulture
"en-US"
CurrentUICulture
"en-US"
RecurringJobId
"TimesheetSyncTask"
RetryCount
10

State

9/1/2023 3:52:50 PM (+1m 34.810s) Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+3.524s Processing

Server:
ec2amaz-sbdjgq5.default:2272
Worker:
90623c3e

+1h 52m Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+7ms Scheduled

Retry attempt 10 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 3:51:11 PM

+1m 34.476s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+67ms Processing

Server:
ec2amaz-sbdjgq5.default:3872
Worker:
3149312a

+1h 9m Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+6ms Scheduled

Retry attempt 9 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 1:57:08 PM

+1m 18.550s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+14.824s Processing

Server:
ec2amaz-sbdjgq5.default:1424
Worker:
1b3d6a35

+44m 750ms Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+6ms Scheduled

Retry attempt 8 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 12:46:45 PM

+1m 31.974s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+28.190s Processing

Server:
ec2amaz-sbdjgq5.default:476
Worker:
dd6621b1

+2.356s Processing

Server:
ec2amaz-sbdjgq5.default:2000
Worker:
665cdc40

+25m 19.717s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+3ms Scheduled

Retry attempt 7 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 12:00:37 PM

+1m 22.674s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+2.723s Processing

Server:
ec2amaz-sbdjgq5.default:4420
Worker:
f9075f5d

+12m 50.073s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+4ms Scheduled

Retry attempt 6 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 11:33:47 AM

+1m 22.143s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+43ms Processing

Server:
ec2amaz-sbdjgq5.default:2884
Worker:
9808fa43

+5m 39.117s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+3ms Scheduled

Retry attempt 5 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 11:19:42 AM

+1m 21.004s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+43ms Processing

Server:
ec2amaz-sbdjgq5.default:2884
Worker:
9808fa43

+2m 21.103s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+4ms Scheduled

Retry attempt 4 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 11:12:44 AM

+1m 23.953s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+37ms Processing

Server:
ec2amaz-sbdjgq5.default:2884
Worker:
bd29bf6b

+1m 47.056s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+4ms Scheduled

Retry attempt 3 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 11:08:50 AM

+1m 24.943s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+3.077s Processing

Server:
ec2amaz-sbdjgq5.default:2884
Worker:
9808fa43

+51.680s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+13ms Scheduled

Retry attempt 2 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 11:05:31 AM

+1m 22.520s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+847ms Processing

Server:
ec2amaz-sbdjgq5.default:2884
Worker:
bd29bf6b

+28.446s Enqueued

Triggered by DelayedJobScheduler

Queue:
default

+27ms Scheduled

Retry attempt 1 of 10: Access Denied: The Personal Access Token used has…

Enqueue at:
9/1/2023 11:03:24 AM

+1m 47.293s Failed

An exception occurred during performance of the job.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException

Access Denied: The Personal Access Token used has expired.

Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: Access Denied: The Personal Access Token used has expired.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__47`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationsAsync>d__64.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<EnsureResourceLocationsPopulated>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<GetResourceLocationAsync>d__63.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<CreateRequestMessageAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__28`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<Query>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 53
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at TeamServices.WorkItemMaterializer.<<Query>b__5_0>d.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemMaterializer.cs:line 37
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at TeamServices.WorkItemRepository.<FindByReference>d__6`1.MoveNext() in C:\TeamServices\Agent\_work\395\s\TeamServices\WorkItemRepository.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServicesImpl.cs:line 16
   at System.Reactive.ExceptionHelpers.ThrowIfNotNull(Exception exception) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Internal\ExceptionServices.cs:line 20
   at System.Reactive.Subjects.AsyncSubject`1.GetResult() in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Subjects\AsyncSubject.cs:line 437
   at Application.Sync.TimesheetSyncTask.<Execute>d__6.MoveNext() in C:\TeamServices\Agent\_work\395\s\Application\Sync\TimesheetSyncTask.cs:line 56

+12h 1m Processing

Server:
ec2amaz-sbdjgq5.priority:2884
Worker:
8e569c35

+11.670s Processing

Server:
ec2amaz-sbdjgq5.priority:2248
Worker:
3d18fe8c

+10ms Enqueued

Triggered by recurring job scheduler

Queue:
priority

8/31/2023 11:00:00 PM Created