T11 Hypercare review

Conversation (turn by turn)

1. turn 1 · 0:27 · 0:27 Ana Duarte 2025-03-18

   Two and a half weeks live. This is the call where we decide whether, to come off daily hypercare.

2. turn 2 · 0:39 · 0:39 Priya Nair 2025-03-18

   Mm-hm.

3. turn 3 · 0:40 · 0:40 Lena Fischer 2025-03-18

   Before that,

4. turn 4 · 0:45 · 0:45 Tomas Lindholm 2025-03-18

   Mm, okay.

5. turn 5 · 0:46 · 0:46 Lena Fischer 2025-03-18

   I want

6. turn 6 · 0:53 · 0:53 Ana Duarte 2025-03-18

   Yeah, yeah.

7. turn 7 · 0:57 · 0:57 Lena Fischer 2025-03-18

   want to talk about the feed.

8. turn 8 · 1:04 · 1:04 Tomas Lindholm 2025-03-18

   Sure.

9. turn 9 · 1:07 · 1:07 Ana Duarte 2025-03-18

   Go ahead.

10. turn 10 · 1:13 · 1:13 Lena Fischer 2025-03-18

    you know It has failed twice. Once on the sixth, and once,

11. turn 11 · 1:24 · 1:24 Tomas Lindholm 2025-03-18

    Okay.

12. turn 12 · 1:28 · 1:28 Lena Fischer 2025-03-18

    on the fourteenth. Both, DC-2. On the fourteenth my

13. turn 13 · 1:38 · 1:38 Ana Duarte 2025-03-18

    Yes.

14. turn 14 · 1:40 · 1:40 Lena Fischer 2025-03-18

    my planners had yesterday's numbers all morning and nobody told them.

15. turn 15 · 1:51 · 1:51 Tomas Lindholm 2025-03-18

    Yep.

16. turn 16 · 1:54 · 1:54 Priya Nair 2025-03-18

    The monitoring fired.

17. turn 17 · 2:01 · 2:01 Tomas Lindholm 2025-03-18

    1 second Mm.

18. turn 18 · 2:02 · 2:02 Lena Fischer 2025-03-18

    The monitoring fired at Kwame, who is in Portugal.

19. turn 19 · 2:10 · 2:10 Ana Duarte 2025-03-18

    That is,

20. turn 20 · 2:16 · 2:16 Lena Fischer 2025-03-18

    Yeah.

21. turn 21 · 2:17 · 2:17 Ana Duarte 2025-03-18

    my fault. I raised the cover question in January and I did not close it.

22. turn 22 · 2:24 · 2:24 Lena Fischer 2025-03-18

    Yeah, yeah.

23. turn 23 · 2:28 · 2:28 Lena Fischer 2025-03-18

    actually I am not looking for someone to blame, I am telling you what happened. The...

24. turn 24 · 2:37 · 2:37 Tomas Lindholm 2025-03-18

    Right.

25. turn 25 · 2:38 · 2:38 Lena Fischer 2025-03-18

    order proposals on the fourteenth were built on

26. turn 26 · 2:43 · 2:43 Ana Duarte 2025-03-18

    True.

27. turn 27 · 2:46 · 2:46 Lena Fischer 2025-03-18

    stale stock, so we over-ordered on about forty lines,

28. turn 28 · 2:51 · 2:51 Ana Duarte 2025-03-18

    Yes.

29. turn 29 · 2:55 · 2:55 Lena Fischer 2025-03-18

    and two store managers have

30. turn 30 · 3:01 · 3:01 Tomas Lindholm 2025-03-18

    1 second Mm-hm.

31. turn 31 · 3:05 · 3:05 Lena Fischer 2025-03-18

    now told their regional manager that the new system does not work.

32. turn 32 · 3:13 · 3:13 Tomas Lindholm 2025-03-18

    That is the expensive part. Not the forty lines.

33. turn 33 · 3:18 · 3:18 Lena Fischer 2025-03-18

    Yes.

34. turn 34 · 3:21 · 3:21 Ana Duarte 2025-03-18

    What do you need from

35. turn 35 · 3:26 · 3:26 Lena Fischer 2025-03-18

    Yes.

36. turn 36 · 3:28 · 3:28 Ana Duarte 2025-03-18

    us?

37. turn 37 · 3:33 · 3:33 Lena Fischer 2025-03-18

    Somebody who is not on holiday.

38. turn 38 · 3:38 · 3:38 Priya Nair 2025-03-18

    Yeah.

39. turn 39 · 3:39 · 3:39 Ana Duarte 2025-03-18

    Tomas is covering until Kwame is back on Monday. From then I want a named second person on...

40. turn 40 · 3:49 · 3:49 Tomas Lindholm 2025-03-18

    Right.

41. turn 41 · 3:53 · 3:53 Ana Duarte 2025-03-18

    the runbook and I will get that agreed internally and then the...

42. turn 42 · 4:05 · 4:05 Lena Fischer 2025-03-18

    Yeah.

43. turn 43 · 4:07 · 4:07 Lena Fischer 2025-03-18

    Thank you.

44. turn 44 · 4:12 · 4:12 Priya Nair 2025-03-18

    Yeah.

45. turn 45 · 4:13 · 4:13 Priya Nair 2025-03-18

    Can I add the technical version, sort of because... matters. The, failure on the sixth was the middleware. The failure on the fourteenth was... not. On the fourteenth and...

46. turn 46 · 4:38 · 4:38 Tomas Lindholm 2025-03-18

    Understood.

47. turn 47 · 4:40 · 4:40 Priya Nair 2025-03-18

    files, and one of the three files was empty because an upstream export

48. turn 48 · 4:49 · 4:49 Tomas Lindholm 2025-03-18

    Right.

49. turn 49 · 4:52 · 4:52 Priya Nair 2025-03-18

    had not finished. So we now have two...

50. turn 50 · 4:56 · 4:56 Lena Fischer 2025-03-18

    Yeah.

51. turn 51 · 4:59 · 4:59 Priya Nair 2025-03-18

    One second. different failure modes and only one of them is the thing we have been discussing since April.

52. turn 52 · 5:20 · 5:20 Tomas Lindholm 2025-03-18

    Sure.

53. turn 53 · 5:23 · 5:23 Tomas Lindholm 2025-03-18

    Is the empty file detectable?

54. turn 54 · 5:30 · 5:30 Lena Fischer 2025-03-18

    Sure.

55. turn 55 · 5:33 · 5:33 Priya Nair 2025-03-18

    Trivially. Nobody

56. turn 56 · 5:38 · 5:38 Ana Duarte 2025-03-18

    Mm, okay.

57. turn 57 · 5:42 · 5:42 Priya Nair 2025-03-18

    was checking file size.

58. turn 58 · 5:46 · 5:46 Lena Fischer 2025-03-18

    Yeah.

59. turn 59 · 5:48 · 5:48 Tomas Lindholm 2025-03-18

    Then we check file I mean...

60. turn 60 · 5:52 · 5:52 Priya Nair 2025-03-18

    Yes.

61. turn 61 · 5:55 · 5:55 Tomas Lindholm 2025-03-18

    Yeah.

62. turn 62 · 5:59 · 5:59 Ana Duarte 2025-03-18

    Both go in the incident log with the distinction kept, please, so because in six months somebody will merge them into one problem.

63. turn 63 · 6:17 · 6:17 Tomas Lindholm 2025-03-18

    Yeah.

64. turn 64 · 6:18 · 6:18 Priya Nair 2025-03-18

    I will

65. turn 65 · 6:25 · 6:25 Lena Fischer 2025-03-18

    Mm.

66. turn 66 · 6:26 · 6:26 Priya Nair 2025-03-18

    write it up.

67. turn 67 · 6:31 · 6:31 Ana Duarte 2025-03-18

    Okay so. On hypercare. Two failures in two and a half,

68. turn 68 · 6:42 · 6:42 Priya Nair 2025-03-18

    Yeah.

69. turn 69 · 6:45 · 6:45 Ana Duarte 2025-03-18

    weeks, both recovered, no data loss.

70. turn 70 · 6:49 · 6:49 Lena Fischer 2025-03-18

    Sure.

71. turn 71 · 6:53 · 6:53 Ana Duarte 2025-03-18

    My recommendation is twice weekly for another three weeks, then weekly.

72. turn 72 · 6:59 · 6:59 Lena Fischer 2025-03-18

    With the cover problem fixed.

73. turn 73 · 7:06 · 7:06 Tomas Lindholm 2025-03-18

    Right.

74. turn 74 · 7:08 · 7:08 Ana Duarte 2025-03-18

    With the cover problem fixed.

75. turn 75 · 7:18 · 7:18 Lena Fischer 2025-03-18

    Then agreed.

76. turn 76 · 7:22 · 7:22 Tomas Lindholm 2025-03-18

    Sure.

77. turn 77 · 7:26 · 7:26 Tomas Lindholm 2025-03-18

    One more thing, not urgent. The planners in two of the pilot pilot stores

78. turn 78 · 7:40 · 7:40 Ana Duarte 2025-03-18

    Yeah.

79. turn 79 · 7:41 · 7:41 Tomas Lindholm 2025-03-18 · cited by dc2-march-impact

    have started keeping their own spreadsheets again.

80. turn 80 · 7:48 · 7:48 Priya Nair 2025-03-18

    Fair enough.

81. turn 81 · 7:52 · 7:52 Lena Fischer 2025-03-18

    I know. I have seen them.

82. turn 82 · 7:59 · 7:59 Tomas Lindholm 2025-03-18 · cited by dc2-trust-event

    That usually happens after a trust event rather than

83. turn 83 · 8:07 · 8:07 Priya Nair 2025-03-18

    Mm.

84. turn 84 · 8:09 · 8:09 Tomas Lindholm 2025-03-18

    than because the

85. turn 85 · 8:15 · 8:15 Priya Nair 2025-03-18

    True.

86. turn 86 · 8:16 · 8:16 Tomas Lindholm 2025-03-18

    numbers are wrong.

87. turn 87 · 8:23 · 8:23 Lena Fischer 2025-03-18

    Sure.

88. turn 88 · 8:26 · 8:26 Lena Fischer 2025-03-18 · cited by dc2-march-impact

    It started on the fifteenth. The day after.

89. turn 89 · 8:34 · 8:34 Priya Nair 2025-03-18

    Yeah.

90. turn 90 · 8:37 · 8:37 Tomas Lindholm 2025-03-18 · cited by dc2-trust-event

    Then it is a trust event, and it will not fix itself by the numbers getting better.

91. turn 91 · 8:53 · 8:53 Lena Fischer 2025-03-18

    What fixes it?

92. turn 92 · 8:56 · 8:56 Tomas Lindholm 2025-03-18

    Yes.

93. turn 93 · 9:00 · 9:00 Tomas Lindholm 2025-03-18 · cited by dc2-trust-event

    Time, you know and them not being surprised again.

94. turn 94 · 9:09 · 9:09 Lena Fischer 2025-03-18

    Sorry, what? Then let us so the...

95. turn 95 · 9:20 · 9:20 Tomas Lindholm 2025-03-18 · cited by dc2-premise-wrong

    One thing before we stop, and it is mine to correct. At cutover I said a file size check would catch this class of failure.

96. turn 96 · 9:33 · 9:33 Priya Nair 2025-03-18

    It would not have caught either of these.

97. turn 97 · 9:37 · 9:37 Tomas Lindholm 2025-03-18 · cited by dc2-premise-wrong,dc2-premise-wrong

    No. It would not. Both files were the size I would have expected. The premise was wrong. A file can be the right size and still be the wrong file. I would withdraw that recommendation.

98. turn 98 · 9:57 · 9:57 Ana Duarte 2025-03-18

    It is in the cutover pack.

99. turn 99 · 10:01 · 10:01 Tomas Lindholm 2025-03-18 · cited by dc2-premise-wrong

    1 second It is in the cutover pack, the runbook and the hypercare plan, because I wrote it once and it got copied. Nobody is going to go back and take it out of three documents.

100. turn 100 · 10:19 · 10:19 Lena Fischer 2025-03-18

     So what replaces it?

101. turn 101 · 10:22 · 10:22 Tomas Lindholm 2025-03-18

     Something that looks inside the file. I do not have it today.

102. turn 102 · 10:29 · 10:29 Priya Nair 2025-03-18

     Mm. We will keep the size one in the meantime, because it is there.