| View previous topic :: View next topic |
| Author |
Message |
njepsen
Joined: 13 Aug 2007
Posts: 469
|
 Posted: Mon Dec 07, 2020 5:20 am Post subject: Kill large number of files |
 |
|
I have a situation where i need to kill a large number of files (upwards of 10,000) on an SD card.
This is done remotely by sms, so I cant just put the SD into a PC
The following code is too slow - it seems to take about 1sec per file.
rmdir doesn't appear to work if the dir is not empty.
Any suggestions?
| Code: |
$regfile = "m1284pdef.dat"
$crystal = 9830400
$framesize =1500
$hwstack = 400 '
$swstack = 400
$frameprotect = 1
config submode = old 'old = need to declare subs or functions
$include "config_mmcsd_hc.bas" 'set up the high capacity SD card comms
$include "config_AVR-DOS.BAS"
$PROG &HFF,&HC7,&HD0,&HFE
'****DELETE CACHE ************************************************************
'Delete all files in the cache on SD card
local TD as long
local filecountw as word
If Instr(inward_sms , "deletecache") <> 0 Then
Inward_sms = ""
Chdir "\"
Chdir "cache"
TD = local_secl + 300 '5 minute timeout
myfilename = dir("*.*") 'find first file.
filecountw=0
do
if len(myfilename) > 0 then
incr filecountw
kill myfilename
else
print #1,">> no files left to kill ";filecountw;" files killed"
exit do 'no files left so exit
end if
myfilename = dir() 'find next file.
loop until local_secL > TD 'safety timeout
Chdir "\"
tempstr1 = "deleted cache," + str(filecountw) +" files"
Call Sendsms( tempstr1 , Sms_sender)
End If
|
(BASCOM-AVR version : 2.0.8.2 , Latest : 2.0.8.3 )
_________________
Neil |
|
| Back to top |
|
 |
laborratte
Joined: 27 Jul 2005
Posts: 299
Location: Berlin
|
| Posted: Mon Dec 07, 2020 11:12 am Post subject: |
|
|
What do you expect? Keep in mind that DOS has to find the FAT entries for every file and mark them as free, same for the dir entries.
Killing a dir with 10,000 files took on my PC with a 4-core 64Bit-CPU @ 3,4GHZ, 16GB RAM and a SSD with 3,500 MB/sec read and 2,100 MB/sec write performance about 4,5 seconds.
So, on a 8-Bit-CPU @ 9,8Mhz, 16KB RAM and SD-Card with maybe 80KB/sec... 3-4h is not that bad.
Maybe you think about a complete different way of whatever-you-caching.... |
|
| Back to top |
|
|
Evert :-)
Joined: 18 Feb 2005
Posts: 2156
|
|
| Back to top |
|
|
EDC
Joined: 26 Mar 2014
Posts: 971
|
| Posted: Mon Dec 07, 2020 3:10 pm Post subject: |
|
|
Maybe plan those killings. 
Simply kill the first file and you are ready to write into it again "fresh" data. Then set the kill_flag and in the next loop kill slowly the rest of the files.
After the last file delete flag and you are done.
No hurry |
|
| Back to top |
|
|
MWS
Joined: 22 Aug 2009
Posts: 2262
|
| Posted: Mon Dec 07, 2020 8:41 pm Post subject: Re: Kill large number of files |
|
|
| njepsen wrote: | | The following code is too slow - it seems to take about 1sec per file. |
I'd check what slows it down, whether it is
or
If dir() takes a bigger part, then leave it out.
Your software likely did create the files, thus it knows the name scheme and can iterate through them.
Calculate the potential names again and try a kill() on each, gDOSError should tell about whether a kill was successful.
If a few successive DOSErrors occur, you can consider it mostly done, the rest of cleaning up can be done by the dir()/kill() method.
However, if kill() on a non-existing file takes longer than a dir(), this approach is screwed.
But - you wanted new ideas, and that's one. |
|
| Back to top |
|
|
EDC
Joined: 26 Mar 2014
Posts: 971
|
| Posted: Wed Dec 09, 2020 4:09 pm Post subject: |
|
|
If you realy look what is happen when you Delete file then you realize that this area of FAT is marked as "permitted to write" and your file is only marked "as erased".
but it is steel there. It is only marked as deleted and space is permitted to be written.
Files are newer deleted. Only space is marked as free but this handle Card controller.
(yes I wrote this twice or more times  )
So I can read back your pictures from SD card when you delete all the pictures..cause they are only marked as deleted.
Obfuscating the previous writing relies on writing random values on the places marked as "empty".
So.. deleting not take more time... the data is steel there. Better think to reset your file counter and write into File000->+ whitout append because normal open causes overwrite. |
|
| Back to top |
|
|
MWS
Joined: 22 Aug 2009
Posts: 2262
|
| Posted: Wed Dec 09, 2020 9:15 pm Post subject: |
|
|
| I give you a still and a never. |
|
| Back to top |
|
|
MWS
Joined: 22 Aug 2009
Posts: 2262
|
| Posted: Thu Dec 10, 2020 10:44 am Post subject: Re: Kill large number of files |
|
|
| njepsen wrote: | | I have a situation where i need to kill a large number of files (upwards of 10,000) on an SD card. |
If:
a) 'kill large file numbers' can be equaled with kill 'em all = format
b) some flash is free
c) the linked code actually works
then this may be another solution:
https://www.ulrichradig.de/forum/viewtopic.php?f=16&t=448
It's C-code, but easy to translate, MBR[512] can be done with Data statements, DriveWriteSector() replaces mmc_write_sector(), and Buffer[512] already exists in AVR-Dos, u08 is like 'Dim as byte'.
As far I understood the code, it's a bit the kind of crowbar type, it may be more subtle to actually build up the FAT according it's structure requirements (this may save some flash too), than simply banging a clean copied FAT into the card.
In case it works, it may be a simple and fast solution to flush thousands of files down the drain. |
|
| Back to top |
|
|
albertsm
Joined: 09 Apr 2004
Posts: 5921
Location: Holland
|
| Posted: Thu Dec 10, 2020 12:07 pm Post subject: |
|
|
if you want to 'format' you could also check out the bascom demo that uses a ram disk for simulating avr-dos.
but as you asked to kill lots of files and not how to format i wonder if that is the solution you need.
_________________
Mark |
|
| Back to top |
|
|
laborratte
Joined: 27 Jul 2005
Posts: 299
Location: Berlin
|
| Posted: Thu Dec 10, 2020 4:51 pm Post subject: |
|
|
About formatting:
IMHO it is not a good idea to write a hard coded fixed MBR to a SD-Card as shown in the C-Code example. This can ruin the card if physical parameters (i.e. size) are not identical. AVR-DOS gives the needed informations after filesysteminit():
glFATFirstSector (Long): Number of first Sector of FAT Area on the Card
gbNumberOfFATs (Byte): Count of FAT copies
gwSectorsPerFat (Word): Count of Sectors per FAT
glRootFirstSector (Long): Number of first Sector of Root Area on the Card
gbSectorsPerCluster (Byte): Count of Sectors per Cluster
gbFilesystemStatus.0 (Bit): FAT16 (=0) or FAT32 (=1)
With the two low-level drive access routines DRIVEREADSECTOR(wSRAMPointer, lSectorNumber) and DRIVEWRITESECTOR(wSRAMPointer, lSectorNumber) it should possible to format a SD-Card like following pseudo code:
| Code: | 'clear 512 Byte of RAMBuffer
'write RAMBuffer to all FAT sectors
For I = 0 to gbNumberOfFATs - 1
For J = 0 to gwSectorsPerFat - 1
lSectorNumber = glFATFirstSector + I*gwSectorsPerFat + J
DRIVEWRITESECTOR (RAMBuffer, lSectorNumber)
Next
Next
'write RAMBuffer to 1st root dir sector
DRIVEWRITESECTOR (RAMBuffer, glRootFirstSector)
'write content to 1st FAT sector
if FAT16 then
'write (FF,FF,FF,F0) at start in RAMBuffer
else
'write (F0,FF,FF,F0,FF,FF,FF,F0) at start in RAMBuffer
end if
For I = 0 to gbNumberOfFATs - 1
lSectorNumber = glFATFirstSector + I*gwSectorsPerFat
DRIVEWRITESECTOR (RAMBuffer, lSectorNumber)
Next
if FAT32 then
'root dir is in data space, so mark corresponding cluster as occupied
'calculate cluster# from glRootFirstSector (keep in mind that there is an offset of two clusters in FAT)
'calculate FATSector from cluster#
'calculate offset from cluster#
DRIVEREADSECTOR (RAMBuffer, FATSector)
'write (FF,FF,FF,F0) at offset in RAMBuffer
DRIVEWRITESECTOR (RAMBuffer, FATSector)
end if
|
|
|
| Back to top |
|
|
MWS
Joined: 22 Aug 2009
Posts: 2262
|
| Posted: Thu Dec 10, 2020 4:58 pm Post subject: |
|
|
| albertsm wrote: | | but as you asked to kill lots of files and not how to format i wonder if that is the solution you need. |
To format was only my suggestion, which arose from the thought, that in case a clogged up SD-card needs to be cleaned up remotely, what other option is available, if no format command exists?
Right, the only feasible option is to kill file by file, which takes lots of time.
With format the structure is cleaned without a need of iterating through all files.
If he wants to keep 3 files and deletes 9997, then my suggestion is useless until he finds a solution to buffer the 3 files. |
|
| Back to top |
|
|
njepsen
Joined: 13 Aug 2007
Posts: 469
|
| Posted: Mon Dec 14, 2020 10:09 pm Post subject: |
|
|
Sorry I have replied to you people earlier, and thanks for the input. Ive been away for a few days.
MWS - I don't need to keep anything on the Sd so format would be OK, but there is no format cmd.
The question you are asking is how did they get there in the first place:
This is a data logger and if the logger loses coms with the cellular network, then the files are stored to the SD; the client left the logger running in his basement for a few months, with no cellular connection, so it faithfully logged data every 15 minutes to the SD.
Now it has to go back in service and has cellular coms at the new location, so of course it is busy sending random useless data down the pipe.
I solved the problem by getting the client to drive to site & swap out the sd. But it would be nice to be able to do it remotely.
_________________
Neil |
|
| Back to top |
|
|
MWS
Joined: 22 Aug 2009
Posts: 2262
|
| Posted: Tue Dec 15, 2020 9:02 am Post subject: |
|
|
| njepsen wrote: | | format would be OK |
My guess.
| Quote: | | but there is no format cmd. |
It doesn't need to stay this way.
I'm no AVR-Dos specialist, so I looked up the brute force method.
However, if 'Laborratte's code is correct, it's a cleaner and flash saving solution, I'd go for it. |
|
| Back to top |
|
|
|
FAQ
Search
Register
Log in
