Quoting: BlackBloodRumDid you create the directory /mnt/data3?

$ mkdir /mnt/data
$ mount /dev/data/root /mnt/data

*Facepalm*

Let's just pretend I didn't ask that...

See? This is what happens when you're rushing to muck around with the computer before you've gotta take off for work; you miss crucial steps.

Anyway, I've managed to locate my data - seemingly intact - and I'm going to (manually) back it up somewhere tonight, before "clean" installing Pop!_OS... But this leaves two questions.

Firstly, what actually happened? Why can I not enter my encryption password when booting into my "normal" (installed) distro, but can access everything from within a "live" distro as if nothing is wrong?

And secondly, what can I do to prevent this from occurring in the future? Not encrypt my storage drive during installation?

Quoting: Cyba.Cowboy

Quoting: BlackBloodRumDid you create the directory /mnt/data3?

$ mkdir /mnt/data
$ mount /dev/data/root /mnt/data

*Facepalm*

Let's just pretend I didn't ask that...

See? This is what happens when you're rushing to muck around with the computer before you've gotta take off for work; you miss crucial steps.

Anyway, I've managed to locate my data - seemingly intact - and I'm going to (manually) back it up somewhere tonight, before "clean" installing Pop!_OS... But this leaves two questions.

Firstly, what actually happened? Why can I not enter my encryption password when booting into my "normal" (installed) distro, but can access everything from within a "live" distro as if nothing is wrong?

And secondly, what can I do to prevent this from occurring in the future? Not encrypt my storage drive during installation?

Glad you got to your data!

As for the causation, we can safely rule out the encryption itself as the root cause in this case, if the LUKS encryption was broken you wouldn't be able to mount it anywhere and would likely need to restore the LUKS header to fix it. (You backed that up right? ) In this case, it appears not to be the issue. It's also not the LVM within, as we can see that has loaded just fine and in read/write mode (good).

It is possible to be slight filesystem corruption from a bad disk. But, I don't think that is the case here.

I'm leaning on it being the kernel here. Did you do updates before you shut down by any chance? A bad kernel update can cause devices (such as your keyboard) to do strange things.

The kernel is the first thing your system loads before even unlocking LUKS. It could well cause the odd input issues you saw with a faulty module (driver) for your keyboard.

The live distro is probably running a different kernel version and thus not affected.

As for prevention - I always suggest keeping two kernel versions installed until you're absolutely certain the latest kernel functions as necessary, this way you can just drop back to the old one if something goes kaput in the new one. You don't want to be stuck with only the latest kernel that happens to not work.

Oh, and of course: Always keep a backup of your data and test it!

However; for your own peace of mind it might be worth checking the smart data of your disk and look for any warning signs, for example:

$ smartctl --all /dev/sda

If there's no obvious warning signs, it might be worth running a quick smart check:
$ smartctl -t short /dev/sda
(Replace sda with your disks letter, do not include the partition numbers.)

Give it a couple of minutes and you should see the test result with:

$ smartctl -l selftest /dev/sda
(In progress test may not show on some disks, may have to wait - you can check the progress in that case with --all)

With a bit of luck, everything will be fine.

Anyhow, I'm rambling, sorry!

Quoting: BlackBloodRumAs for prevention - I always suggest keeping two kernel versions installed until you're absolutely certain the latest kernel functions as necessary, this way you can just drop back to the old one if something goes kaput in the new one. You don't want to be stuck with only the latest kernel that happens to not work.

I think Pop_OS already does this.

AFAICT it uses systemd-boot and pressing space during boot should allow you to select the previously installed kernel and a recovery option:

https://support.system76.com/articles/pop-recovery/

Last edited by whizse on 1 November 2024 at 2:30 pm UTC

You won't believe this, but after all that mucking around, my laptop is working perfectly now... As if nothing had happened ( I thought I'd check it one last time, prior to a "clean" install).

Insert a tirade of swearing

Quoting: BlackBloodRumwould likely need to restore the LUKS header to fix it. (You backed that up right? )

I just manually copied the data to an external storage drive, so the data is not encrypted in the storage drive...

Quoting: BlackBloodRumI'm leaning on it being the kernel here. Did you do updates before you shut down by any chance? A bad kernel update can cause devices (such as your keyboard) to do strange things.

I only removed Geary... But as I said above, I foolishly didn't read what was being removed when I "purged" Geary and just hit "Y", before shutting down shortly thereafter - it's possible that something important was being removed.

I had updates / upgrades there to install, but I don't recall installing anything, as I typically do this on the 1st of the month (in case any bad updates / upgrades are pushed out).

I could be wrong of course, it's a week or two ago now and my memory is not what it used to be... But I'm pretty sure I didn't do any updates / upgrades prior to shutting down.


Here's what came up with a 'SMART' check, by the way:
=== START OF INFORMATION SECTION ===
Model Number:                       WD PC SN560 SDDPNQE-1T00-1002
Serial Number:                      2348SB402778
Firmware Version:                   74104000
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 1,024,209,543,168 [1.02 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,024,209,543,168 [1.02 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 4a418ad007
Local Time is:                      Sat Nov  2 00:25:29 2024 AEST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x00df):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp Verify
Log Page Attributes (0x7e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg *Other*
Maximum Data Transfer Size:         256 Pages
Warning  Comp. Temp. Threshold:     84 Celsius
Critical Comp. Temp. Threshold:     88 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     5.00W    4.20W       -    0  0  0  0        0       0
 1 +     3.20W    3.20W       -    0  0  0  0        0       0
 2 +     2.30W    2.30W       -    0  0  0  0        0       0
 3 -   0.0150W       -        -    3  3  3  3     1500    2500
 4 -   0.0050W       -        -    4  4  4  4    10000    6000
 5 -   0.0033W       -        -    5  5  5  5   176000   25000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    1,543,187 [790 GB]
Data Units Written:                 2,542,334 [1.30 TB]
Host Read Commands:                 12,282,210
Host Write Commands:                24,475,203
Controller Busy Time:               221
Power Cycles:                       68
Power On Hours:                     37
Unsafe Shutdowns:                   38
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               46 Celsius
Temperature Sensor 2:               34 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

I'm curious as to what it considers an "unsafe" shutdown - I deliberately wait for the timer to end when I shut down, believing that it will have safely disconnected all filesystems, closed all programs, etc by the time the computer is actually shut down.

Last edited by Cyba.Cowboy on 1 November 2024 at 2:35 pm UTC

Quoting: Cyba.CowboyYou won't believe this, but after all that mucking around, my laptop is working perfectly now... As if nothing had happened ( I thought I'd check it one last time, prior to a "clean" install).

I think we're back to the intermittent stuck key hypothesis now. If it was an upgrade problem it wouldn't have resolved itself.

I gave it a try on my LUKS encrypted laptop and I can more or less reproduce the behavior. When keeping either the volume down button (but not volume up?) or any of the F* keys down, plymouth exits, I get a string of ^[[26~ on screen followed by a terminal prompt that does not recognize my passphrase because it keeps reading the key presses from the foreign key...

Quoting: whizseI think we're back to the intermittent stuck key hypothesis now.

I still think you're wrong, but I have a tool called 'ScreenKey', which for a short time, shows keys being depressed as on-screen text (e.g. "Enter", <--, Tab, volume up, etc) or the command it's trying to use (e.g. XF86Launch3)... So I might leave it running for a few days and see if if detects any unexpected key presses, just to rule out your theory completely (in my experience, it identifies all but one key on my laptop).

Last edited by Cyba.Cowboy on 1 November 2024 at 3:08 pm UTC

Quoting: Cyba.CowboyHere's what came up with a 'SMART' check, by the way:
=== START OF INFORMATION SECTION ===
Model Number:                       WD PC SN560 SDDPNQE-1T00-1002
Serial Number:                      <Snip>
Firmware Version:                   74104000
PCI Vendor/Subsystem ID:            0x15b7
IEEE OUI Identifier:                0x001b44
Total NVM Capacity:                 1,024,209,543,168 [1.02 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,024,209,543,168 [1.02 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            001b44 4a418ad007
Local Time is:                      Sat Nov  2 00:25:29 2024 AEST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x00df):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp Verify
Log Page Attributes (0x7e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg *Other*
Maximum Data Transfer Size:         256 Pages
Warning  Comp. Temp. Threshold:     84 Celsius
Critical Comp. Temp. Threshold:     88 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     5.00W    4.20W       -    0  0  0  0        0       0
 1 +     3.20W    3.20W       -    0  0  0  0        0       0
 2 +     2.30W    2.30W       -    0  0  0  0        0       0
 3 -   0.0150W       -        -    3  3  3  3     1500    2500
 4 -   0.0050W       -        -    4  4  4  4    10000    6000
 5 -   0.0033W       -        -    5  5  5  5   176000   25000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    1,543,187 [790 GB]
Data Units Written:                 2,542,334 [1.30 TB]
Host Read Commands:                 12,282,210
Host Write Commands:                24,475,203
Controller Busy Time:               221
Power Cycles:                       68
Power On Hours:                     37
Unsafe Shutdowns:                   38
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               46 Celsius
Temperature Sensor 2:               34 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

I'm curious as to what it considers an "unsafe" shutdown - I deliberately wait for the timer to end when I shut down, believing that it will have safely disconnected all filesystems, closed all programs, etc by the time the computer is actually shut down.

The drive looks fine, it's also quite normal for the unsafe shutdown counter to go up, even if you're safely powering it down. I mean even mine has them:

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.58-drakul] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KINGSTON SA2000M81000G
Serial Number:                      <Snippidy>
Firmware Version:                   S5Z42109
PCI Vendor/Subsystem ID:            0x2646
IEEE OUI Identifier:                0x0026b7
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization:            996,890,836,992 [996 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            0026b7 6840c23465
Local Time is:                      Fri Nov  1 13:22:32 2024 GMT
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     75 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0
 1 +     4.60W       -        -    1  1  1  1        0       0
 2 +     3.80W       -        -    2  2  2  2        0       0
 3 -   0.0450W       -        -    3  3  3  3     2000    2000
 4 -   0.0040W       -        -    4  4  4  4    15000   15000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        35 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    11%
Data Units Read:                    276,658,037 [141 TB]
Data Units Written:                 203,701,929 [104 TB]
Host Read Commands:                 3,317,223,549
Host Write Commands:                2,884,988,126
Controller Busy Time:               25,443
Power Cycles:                       248
Power On Hours:                     34,047
Unsafe Shutdowns:                   43
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
Num  Test_Description  Status                       Power_on_Hours  Failing_LBA  NSID Seg SCT Code
 0   Extended          Completed without error               26360            -     -   -   -    -

Honestly, compared to my drive yours is still a baby (in terms of age).

Quoting: whizse

Quoting: Cyba.CowboyYou won't believe this, but after all that mucking around, my laptop is working perfectly now... As if nothing had happened ( I thought I'd check it one last time, prior to a "clean" install).

I think we're back to the intermittent stuck key hypothesis now. If it was an upgrade problem it wouldn't have resolved itself.

I gave it a try on my LUKS encrypted laptop and I can more or less reproduce the behavior. When keeping either the volume down button (but not volume up?) or any of the F* keys down, plymouth exits, I get a string of ^[[26~ on screen followed by a terminal prompt that does not recognize my passphrase because it keeps reading the key presses from the foreign key...

Agreed, considering it resolved itself that rules out the kernel.

I haven't really screen any unexpected keypresses; but then, I also haven't had the issue again since. Weird.

Anyway, thank you to you all, and @BlackBloodRum in particular, who was ever so patient with me. You guys are awesome!

Last edited by Cyba.Cowboy on 9 November 2024 at 9:02 am UTC