Post

10 followers Follow
2
Avatar

Periodically cleaning the repository with a CLI script

When using a continuous integration system like Jenkins to do automatic deployments in Deployit, your repository may fill up quickly with build versions of your application. You will need some periodical cleanup to keep the Deployit repository at a manageable size.

At XebiaLabs, we use the attached Deployit CLI script to do this job for us. It allows you to delete packages older than a certain number of days and that match a certain text, or that reside in a certain directory. For example, you can tell it to 'delete all SNAPSHOT versions older than 7 days from Applications/PetClinic'. Since deletion is a destructive operation, you may want to revise what is about to be deleted, so we added a 'dry run' option that serves as a safety cap. With this option enabled, the script simply prints the packages that would be deleted, but doesn't actually delete them.

Usage

save the script on a host that has the Deployit CLI installed, and then invoke it with the command:

bin/cli.sh -f [path to cleanup script] [number of days] [verion marker] [dryrun:True or False] [Path in repository]

For example, to delete all PetClinic snapshot versions older than a week:

bin/cli.sh -f cleanup-repository.py 7 SNAPSHOT False Applications/PetClinic

In Jenkins

To have automatic cleanup, we have set up a scheduled job in Jenkins. 

Steps:

  • Install the Deployit CLI on the same server Deployit is running
  • Install the cleanup-repository.py script in the Deployit CLI directory
  • Make sure the Jenkins server can log into to the Deployit host using SSH. (Alternatively, install the CLI on the Jenkins server)
  • Create a Jenkins job, titled 'Clean up Deployit repository'
  • As Build Trigger, use "Build periodically" and put in a pattern like 0 0 \* \* \* that will trigger the cleanup script every day at midnight.
  • As Build step, add an Execute shell command for each application you would like to cleanup.

For example:

ssh user@deployit "deployit-3.9.2-cli/bin/cli.sh -username admin -password XXXXXX -f ~/cleanup-repository.py -q 7 SNAPSHOT False Applications/PetClinic"

This way the cleanup runs unattended, and we can find the logs and status of the jobs in the Jenkins console at our convenience.

Kind regards,

    Hes Siemelink

XebiaLabs Support Answered

Please sign in to leave a comment.

21 comments

0
Avatar

Not sure what is wrong with the link -- it downloads fine for me. What kind of an error do you get, Marcel?

Regards

Andrew

XebiaLabs

XebiaLabs Support 0 votes
0
Avatar

I modified my version by introducing wildcard * for the parameter [version marker] to delete any packages older than [number of days]

Levent Tutar 0 votes
0
Avatar

I don't think the number of days argument is working, regardless the number of days I am entering I always get the same list of packages, is that normal ?

Mohamed El Hossary 0 votes
0
Avatar

Hi Mohamed,

It is normal if the packages that are imported are older than the number of days that you are entering.

* repository.search(String ciType, Calendar before) : List

What you can do?

Create a new package and do the test again. The new package will not be in the resultset since it is new and for example not older than 1 days.

 

The code is working. This is the part of the code:

now = Calendar.getInstance()

now.add(Calendar.DAY_OF_MONTH, int(days) * -1)

print "Base date is ",now.getTime()

allPackages = repository.search('udm.DeploymentPackage', now)

 

Try manually the CLI interface and experiment with repository.search('udm.DeploymentPackage', now) and at the end do

print allPackages

 

 

Levent Tutar 0 votes
0
Avatar

Hello Levent,

I think I understand what was misleading me, I think the timestamp of the package don't change if it was created by a duplicate action, this is why I was getting weird results. This is tricky because I believe the timestamp of the package should change even if it was created by a duplicate.

 

Regards,

Mohamed El Hossary 0 votes
0
Avatar

I have tested it, a newely created package from scratch, doesn't come up in the results, a duplicate package showes in the results although for example I entered 60 days

Mohamed El Hossary 0 votes
0
Avatar

Hi Levent,

Can you post your version with wildcard for the parameter [version marker]  ?

thx for your help, i'm lost with the CLI part...

 

 

matthieu jacquinot 0 votes
0
Avatar

Hi Levent

is there a workaround to manage the duplicate issue ?

 

Thanks!

 

Mohamed

Mohamed El Hossary 0 votes
0
Avatar

Not that I know of. You may submit this as a feature request.

When I use the duplicate function in GUI, and I use the REST API to query the entry. I see that the creation-date is also copied. Furhter more, last-modified tag is also not updated.

http://localhost:4516/deployit/repository/ci/Applications/PetClinic-war/4.0

<udm.DeploymentPackage id="Applications/PetClinic-war/4.0" token="45f00bd0-c122-4d95-a868-9684a322e313" created-by="admin" created-at="2014-10-17T16:07:37.676+0200" last-modified-by="admin" last-modified-at="2014-10-17T16:07:37.676+0200">

http://localhost:4516/deployit/repository/ci/Applications/PetClinic-war/4.0%20Copy

<udm.DeploymentPackage id="Applications/PetClinic-war/4.0 Copy" token="45f00bd0-c122-4d95-a868-9684a322e313" created-by="admin" created-at="2014-10-17T16:07:37.676+0200" last-modified-by="admin" last-modified-at="2014-10-17T16:07:37.676+0200">

BTW. The creation date is not visible through the CLI function deployit.print(XX)

 

 

Levent Tutar 0 votes
0
Avatar

Hi Lokhan,

The file is available as a link when I click on cleanup-repository.py above under the jenkins section.

Levent Tutar 0 votes
1
Avatar

Hi Joris,

The link is not working, could you please provide the latest version using another way ?

Regards

Mohamed El Hossary 1 vote
0
Avatar

Mohamed- The link works for me. I sent the script to you via email.

Regards,

Dan

Daniel Robichau 0 votes
0
Avatar

Not sure if it still works, but here's the script:

```

from java.util import Calendar
import sys

def isWantedApplication(packageId):
if application is None:
return True
if packageId.startswith(application):
return True
return False

if len(sys.argv) < 4:
print "Usage: cleanup.py [days history to keep] [version marker to remove] [dry run?]"
exit(1)

days = sys.argv[1]
versionMarker = sys.argv[2]
dryRun = True
if (sys.argv[3] == "False"):
dryRun = False

if len(sys.argv) == 5:
application = sys.argv[4]
else:
application = None

now = Calendar.getInstance()
now.add(Calendar.DAY_OF_MONTH, int(days) * -1)
print "Base date is ",now.getTime()
allPackages = repository.search('udm.DeploymentPackage', now)
deletedPackage = 0
missDeletedPackage = 0

for packageId in allPackages:
if not isWantedApplication(packageId):
continue
package = repository.read(packageId)
version=packageId.split('/')[-1]
if versionMarker in version:
if bool(dryRun):
print "Dry run - will delete package", package.id
deletedPackage = deletedPackage + 1
else:
print "Deleting package", package.id
try:
repository.delete(package.id)
deletedPackage = deletedPackage + 1
except:
missDeletedPackage = missDeletedPackage +1

print len(allPackages),"\tpackage(s) in the repository imported before", now.getTime()
print deletedPackage,"\tdeleted package(s)"
print missDeletedPackage,"\tcandidate package(s) but not deleted because it's still referenced"
print "done"

```

Joris De Winne 0 votes