Post

4 followers Follow
4
Avatar

Repository export

When moving data from a Deployit installation to another one, you would want to move the contents of your repository as well. Attached is a CLI script for Deployit 3.x that exports the contents of your repository and produces another CLI script which re-creates the data. Note that this script only works for Environment and Infrastructure nodes. Imported binary packages and applications will not be exported.

XebiaLabs Support

Please sign in to leave a comment.

26 comments

0
Avatar

Hello Martin,

This is a really useful script. I will use it as a base to duplicate environments.

Just a note : the password extracted by this script are all initialized with the obfuscated value : "********"

It 's fine for me as it is safer, but one should not forget to re-enter them correctly in the new deployit instance

 

Fabien Arrault 0 votes
0
Avatar

Hi Fabian,

 In Deployit 3.5 password are no longer exported as ******** but as a base64 encoded encrypted value. That allows you to export and copy passwords too. Doesn't help you when using this script to migrate from 3.0 to 3.5 though. :-/

 Regards, Vincent.

XebiaLabs Support 0 votes
0
Avatar

Attached a slightly updated version tested against the Deployit 3.7.2 CLI.

Happy deploying!

 

Andrew

XebiaLabs Support 0 votes
0
Avatar

Very nice script only a pity that the generated script is not written to the current working directory.

Minto van der Sluis 0 votes
0
Avatar

Hi Martin,

Not as easy as you think. In DeployIT 3.7.3 CLI the current working directory 'os.getcwd()' is the location where the cli is installed. Not the location from where the script ran.

For instance:

CLI_HOME = /some/dir/cli/

/some/other/dir>cli -f myscript

I expect my CWD to be /some/other/dir. Bummer! in 'os.getcwd()'  return CLI_HOME instead.

Should I create an issue for this? Or am I doing something wrong here (my second day using python)?

Minto van der Sluis 0 votes
0
Avatar

Hi Minto,

I see what you mean. 

The cli.sh or cli.cmd scripts actually change directory to the CLI installation directory to build up the Java classpath to include the necessary libraries, extensions and hotfixes. If we want to retain the original work directory, we'd have to change the script to refer to absolute paths for the classpath. Depending on the location and number of classpath entries, this could become a problem on platforms that have a maximum length for the command line.

Given that possibility, I like the fixed path the script now uses better. Also, you could point it to a "dropbox" folder where you put all export files instead of in /tmp.

Regards, Martin

XebiaLabs Support 0 votes
0
Avatar

Hi Martin,

We're drifting away from the original subject here so I created Request #435 discussion regarding the original working directory can continue there.

Regards,

Minto

Minto van der Sluis 0 votes
0
Avatar

Hello,

This export script (both version) doesn't work in Deployit 3.8 :

Exception: javax.script.ScriptException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Type  does not contain a prefix in <script> at line number 1

=> How do you ask Deployit 3.8 to search all CI without specifying a ci Type ?

Fabien

Fabien Arrault 0 votes
0
Avatar

Hi Fabien

Thanks for pointing that out. Attached an updated version for 3.8.0, which also exports the Configuration root. Note that 3.8.0 also allows you to export DAR files to move existing deployment packages to a new server:

export-dar-from-ui.png

Regards

Andrew

XebiaLabs

XebiaLabs Support 0 votes
0
Avatar

To export the dar from the CLI :

import os
def exportApplicationDars(exportDir):
if not os.path.exists(exportDir):
os.makedirs(exportDir)
packages = repository.search("udm.DeploymentPackage")
for pkg in packages:
repository.exportDar(exportDir,pkg)

XebiaLabs Support 0 votes
0
Avatar

The repository export script is now available for viewing and downloading from the XebiaLabs community repository.

The script can be invoked as follows:

/path/to/cli/bin/cli -username <user> -password <password> -f /path/to/export.py

and will output a repository-export.py file containing a file that can be run against another Deployit installation to recreate the Infrastructure, Environments and Configuration branches of the exported repository.

Regards

 

Andrew

XebiaLabs

XebiaLabs Support 0 votes
0
Avatar

Hi,

seems a very useful script, but not working, not sure why.   The export.py runs ok, running the repository-export.py as admin at version 3.9.4, I am getting the folowing error:

Reading extension: ext/ci.py

Reading extension: ext/readme.cli

java.io.IOException: Mark invalid

 Could there be a difference between the 3.8.0 repository (export/py last adjustment was for this version) and 3.9.4 ?

Help is appreciated

A.F. van den Heuvel 0 votes
0
Avatar

Hi Joris,

Yes, I am using that version. The export is not the problem, I do get a repository-export.py.

This morning I ran the  repository-export.py in small "chunks" starting with the infrastructureList.append's and the 1st error I now get is on a NotFoundException in the application tree:

On:

infrastructureList.append(create('Infrastructure/Financieren/FB/VZP/O+T/lsrv4676/asa01lsrv4676/Oracle JDBC Driver','was.OracleJdbcProvider',{'providerType':'Oracle JDBC Driver','description':'Oracle JDBC Driver','container':'Infrastructure/Financieren/FB/VZP/O+T/lsrv4676/asa01lsrv4676','implementationClassName':'oracle.jdbc.pool.OracleConnectionPoolDataSource','deployable':'Applications/Financieren/fb/VZP/PEGA_prpc/6.3_SP1_20130905/Oracle JDBC Driver','isolatedClassLoader':'false','classpath':'${ORACLE_JDBC_DRIVER_PATH}/ojdbc6.jar'}))

 

I get a:

2013-12-02 13:28:11.900 [main] INFO  c.x.d.b.r.r.InternalServerErrorClientResponseInterceptor - Handling Error Response, status code: 404

com.xebialabs.deployit.exception.NotFoundException [/repository/cis]: Repository entity [Applications/Financieren/fb/VZP/PEGA_prpc/6.3_SP1_20130905/Oracle JDBC          

 

In the script there are two chunks of create statements:

  • infrastructureList.append(create('Infrastructure
  • environmentsList.append(create('Environments

I don't see any infrastructureList.append(create('Applications

Should there be any create's for the Application tree in the repository ?

 

regards, Ton

 

 

 

 

 

 

 

 

A.F. van den Heuvel 0 votes
0
Avatar

Hi Ton,

What is the size of the exported script.  Is it larger than 100k?  Jython has a maximum limit for scripts.  If your file is greater than 100k, could you try breaking it up into smaller chunks and try importing that way.

 

regards,

Ravan

XebiaLabs Support 0 votes
0
Avatar

I would like to use something like this to build a test environment. Using the github version of this script does not help me sadly enough. I do get an exported file and even added my own application part. After it's exported I wanted to import.

I have seen all kind of errors now. The last one was "Exception in thread "main" javax.script.ScriptException: SyntaxError: no viable

alternative at input 'PFC_CULT' in <script> at line number 32 at column number 1

916". While I tried to import enviroment parts of the tree. (PFC_CULT is a placeholder).

Errors I received before are mark invalid, because the files gets to 2000 kb. So I have to chop it in very small pieces (not workable). After that I got an error on 64bits encrypted passwords, which couldn't be set.

What is the cause of this all? Because I'm running against 3.9.3?

Help would be appreciated.

Thijs van Bloemendaal 0 votes
0
Avatar

Hi ravan,

it's size is 133 KB :(  So that could be an explanation

I did broke it up in smaller chunks, no java.io.IOException: Mark invalid this time, but an invalid reference to a non-existing entry, please see my previous comment in this post about the status code: 404

 

regards, Ton

A.F. van den Heuvel 0 votes
0
Avatar

Ton,

You are running into the limitations of the script as it does not support the exporting of the Application configuration items and the associated DARs.

The export script is also exporting all deployed applications in the system and those have an indirect reference to the application configuration items.

Depending on you use case, you will have to adjust the script accordingly, i.e

  1. If you are only interested in exporting/importing environment and infrastructure, then adjust the script to excluded udm.DeployedApplication types
  2. If you are interested in a complete repository export and import, then you will have to extend the script to also export the Application configuration items as well.  Also you will have to script out exporting/import of the deployment packages (DARs)

regards,

Ravan

XebiaLabs Support 0 votes
0
Avatar

Hi Ravan,

thanks, that explains the 404. I am only interested in importing/exporting the environment/infrastructure objects, so excluding everything else in the script will give me what I need.

Ravan and Joris, thanks for your help,

Regards, Ton

A.F. van den Heuvel 0 votes
0
Avatar

Hi Thijs,

Are you trying to export/import the environment(s)/infrastructure without any applications and deployed applications? If so can you try adjusting the script to exclude any udm.DeployedApplication?

Regards,

Joris

 

XebiaLabs Support 0 votes
0
Avatar

Hi Thijs,

What kind of error do you get when using only infrastructure elements? Can you send us the log output on this?

Regards,

Joris

XebiaLabs Support 0 votes
0
Avatar

Anyone,

We have trouble with the Base64 encoded passwords in the created repository-export.py script

When running the script we get the following error:

Reading extension: ext\ci.py

Reading extension: ext\readme.cli

Saving repository objects

2014-07-16 11:32:31.617 [main] INFO  c.x.d.b.r.r.InternalServerErrorClientResponseInterceptor - Handling Error Response, status code: 500

java.lang.RuntimeException: com.thoughtworks.xstream.converters.ConversionException [/repository/cis]: Could not decrypt Base64 encoded password : Could not decrypt Base64

encoded password ---- Debugging information ---- message             :

Could not decrypt Base64 encoded password cause-exception     :

java.lang.IllegalStateException cause-message       : Could not decrypt

Base64 encoded password class               :

com.xebialabs.deployit.plugin.api.udm.ConfigurationItem required-type

: com.xebialabs.deployit.plugin.api.udm.ConfigurationItem converter-type

: com.xebialabs.deployit.core.rest.xml.XStreamCiConverterWithRepository

path                : /list/overthere.CifsHost/password line number

: 11 class[1]            : java.util.ArrayList converter-type[1]   :

com.xebialabs.xltype.serialization.xstream.ConfigurationItemCollectionConverter

version             : null ------------------------------- Could not

decrypt Base64 encoded password

 

Can somebody help us out?

 

 

 

 

Nico Luitwieler 0 votes
0
Avatar

Hello,

I faced the same issue, the quickest solution I was able to come up with is to replace all Base64 passwords by a text field and to re-enter the passwords again after the import.

It would be nice if someone was able to find a better workaround for this.

 

Regards,

Mohamed El Hossary 0 votes