Search This Blog

Breaking

Tuesday, 29 June 2021

June 29, 2021

How to read paraquet file in java?

 

If you are trying to read paraquet file, then it can be done by adding these 2 dependencies in pom file.






<dependencies>

 <dependency>

 <groupId>org.apache.parquet</groupId>

 <artifactId>parquet-hadoop</artifactId>

 <version>1.9.0</version>

 </dependency>


 <dependency>

 <groupId>org.apache.hadoop</groupId>

 <artifactId>hadoop-common</artifactId>

 <version>2.7.0</version>

 </dependency>

</dependencies>


public class readParaquetFile {

    private static Path path = new Path("C:\\Users\\deepak.mathpal\\Downloads\\userdata1.parquet");
    
    private static void printGroup(Group g) {
        
       int fieldCount = g.getType().getFieldCount();
        
        for (int field = 0; field < fieldCount; field++) {

            int valueCount = g.getFieldRepetitionCount(field);

            Type fieldType = g.getType().getType(field);

            String fieldName = fieldType.getName();

            for (int index = 0; index < valueCount; index++) {

                if (fieldType.isPrimitive()) {

                    System.out.println(fieldName + " " + g.getValueToString(field, index));

                }

            }

        }

        System.out.println("");

    }

    public static void main(String[] args) throws IllegalArgumentException {

        Configuration conf = new Configuration();

        try {

            ParquetMetadata readFooter = ParquetFileReader.readFooter(conf, path, ParquetMetadataConverter.NO_FILTER);

            MessageType schema = readFooter.getFileMetaData().getSchema();

            ParquetFileReader r = new ParquetFileReader(conf, path, readFooter);

            PageReadStore pages = null;

            try {

                while (null != (pages = r.readNextRowGroup())) {

                    final long rows = pages.getRowCount();

                    System.out.println("Number of rows: " + rows);

                    final MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);

                    final RecordReader recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));

                    for (int i = 0; i < rows; i++) {

                        final Object g = recordReader.read();

                        printGroup((Group) g);

                    }

                }

            } finally {

                r.close();

           }

        } catch (IOException e) {

            System.out.println("Error reading parquet file.");

            e.printStackTrace();

        }

    }

}

Friday, 25 June 2021

June 25, 2021

How to get started with TypeScript in Visual Studio code [Windows]?

If you are planning to start doing coding practice in typescript, then the below steps can help you for doing the initial setup.







1. First, install visual studio (windows installer)

2. Install node (windows installer)

3. Check node --version (visual studio new terminal)

4. Once you have a node, you can use npm. So with the help of it, install typescript globally. 

npm install -g typescript (visual studio new terminal)

5. Check the typescript version to make sure that typescript is installed or not. 

tsc --version

Version 4.3.4

or (You might face this issue)

tsc : File C:\Users\deepak.mathpal\AppData\Roaming\npm\tsc.ps1 cannot be loaded because running scripts is disabled on this system. For more information, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170.

At line:1 char:1

+ tsc --version


tsc.cmd --version


6. To fix the above issue, open powershell with admin (visual studio powersheell does not open with admin access)

provide the access:

Open Commandform : windows + R


Type : Powershell


Then type : set-executionpolicy remotesigned


And select opetion : A


7. now, you can run tsc command. 

tsc --version

Version 4.3.4


8. Create a folder and open it in vscode

C:\Users\deepak.mathpal\Documents>mkdir typescript


C:\Users\deepak.mathpal\Documents>cd typescript


C:\Users\deepak.mathpal\Documents\typescript>code .


C:\Users\deepak.mathpal\Documents\typescript>


9. From the File Explorer, create a new file called helloworld.ts.

let message: string = 'Hello World';

console.log(message);


10. To compile your TypeScript code, you can open the Integrated Terminal (Ctrl+`) and type

 tsc helloworld.ts. 

This will compile and create a new helloworld.js JavaScript file.


11. If you have Node.js installed, you can run node helloworld.js.


12. tsconfig.json


So far, we have been relying on the TypeScript compiler's default behavior to compile our TypeScript source code. We can modify the TypeScript compiler options by adding a tsconfig.json file that defines the TypeScript project settings such as the compiler options and the files that should be included.


Add a simple tsconfig.json which set the options to compile to ES6 and use CommonJS modules.13.


13. {

    "compilerOptions": {

      "target": "es6",

      "module": "commonjs",

      "outDir": "out"

    }

  }

  

  14. add above code in tsconfig file, delete the js file eaerlier created, open a new terminal and run again. Now js file will be inside out directory

  

  15. Run code now from out directory.


References:

1. https://stackoverflow.com/questions/58796490/tsc-ps1-cannot-be-loaded-because-running-scripts-is-disabled-on-this-system

2. https://code.visualstudio.com/docs/typescript/typescript-tutorial








Tuesday, 22 June 2021

June 22, 2021

Execute Selenium Script in Already Open browser [Mac OS]

If we want to run selenium execution in an open browser, (in mac machine) we need to first find out our chrome location in mac. Mostly, it remains under the Application folder if you have not done any specific changes while installing the browser. 

Once you get the location, create a new folder at any location and copy its path. 

I created an in-home location. 

Once this is done, run this command in the terminal:


% /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 -user-data-dir=chromedata

It will open a new chrome browser, where we can debug our fix. Now in your automation script, put a option for chromedriver debugging.

 options.setExperimentalOption("debuggerAddress", "localhost:9222");

Now Provide this option to your chromedriver.

In My case, it was something like this:

capabilities.setCapability(ChromeOptions.CAPABILITY, options);

 System.setProperty("webdriver.chrome.driver", chromeDriverPath);

  driver = new ChromeDriver(capabilities);

 Now when you will run your script it will run in this already open browser.

Note: Port number, directory location is optional.

For windows, check this link: Selenium Script execution on already open browser Windows OS

Wednesday, 16 June 2021

June 16, 2021

Execute Selenium Script in Already Open browser [Windows]

 This is a part of my learning series where recent ally I read an article about how to execute selenium script in an already open browser. This is needed when some of our test cases get fails and we put a fix. When we want to test that fix, then our script starts from scratch and this process becomes tedious when our fix doesn't work and we again and again put our fixes. A lot of time spends in unnecessary execution of same steps which are already working.

So, to avoid this situation, and to run our execution exactly from where it's failing, we can use chrome browser native debugging option. 

What we need to do to achieve this is, we need to launch our chrome in debugging mode. Below are the steps which we can follow to achieve this:













1. Find the chorome.exe in your machine. In my case, it is in, C:\Program Files\Google\Chrome\Application.

2. Create a directory in your machine. I created C:\Users\deepak.mathpal\Documents\chromedata

3. Now run this command.

C:\Program Files\Google\Chrome\Application>chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\Users\deepak.mathpal\Documents\chromedata

4. It will open a new chrome browser, where we can debug our fix.

5. Now in your automation script, put a option in your chromedriver for debugging.

 options.setExperimentalOption("debuggerAddress", "localhost:9222");

6. Provide this option to your chromedriver.

In My case, it was something like this:

capabilities.setCapability(ChromeOptions.CAPABILITY, options);

 System.setProperty("webdriver.chrome.driver", chromeDriverPath);

  driver = new ChromeDriver(capabilities);

7. Now when you will run your script it will run in this alredy open browser.


Note: Port number, directory location is optional.

References: 

1. https://chromedevtools.github.io/devtools-protocol/

2. https://www.youtube.com/watch?v=Rrkj4tdXngY




Friday, 11 June 2021

June 11, 2021

Selenium UI Automation for copy clipboard feature

Recently, I came across a UI Automation scenario where I need to validate the clipboard behavior of a field. 

The scenario was: I need to click on a copy SQL button which copies a large SQL which I need to validate. Earlier, I have used the ctrl+c and ctrl+v approach, but here, there was no text field where I can paste that content.

So, at run time, I need to validate the content has been copied or not.  








To automate this, I used the java clipboard class. 

Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();


Once the clipboard object is created, you can click on the element.


driver.findElement(By.cssSelector("div:nth-child(2) > .input-group .octicon-clippy")).click();

Thread.sleep(2000);


Now, you can extract this data.


Object dataFromCopyURLFeature = clipboard.getData(DataFlavor.stringFlavor);

System.out.println("Copied data is " + dataFromCopyURLFeature);


You can assert this data, with the actual data for validation purpose.


Assert.assertTrue(dataFromCopyURLFeature.equals("https://github.com/qamatters/KarateDemo.git"), "Validate the url");


Let me know if code is needed for this.

June 11, 2021

Encrypt and Decrypt username and Password in Automation

Recently, I came across a situation where we need to encrypt usernames and passwords so that they can not be hardcoded. This was my first experience to encounter this kind of situation in automation. So I read some java articles on how can we achieve it. 

I used java cipher, to do this activity. 






In java, You can create an instance for the cipher class and provide a mode that can be used for encryption.

Cipher cipher Cipher.getInstance("AES/ECB/PKCS5Padding");

Before using the cipher obect, you need to initialise it.

cipher.init(Cipher.ENCRYPT_MODEsecretKeySpec);

The first argument is for to set mode (Encrypt/decrypt) and the second argument is to

for creating the secretKeySpec.

The secret key spec you can generate by:

public static void setKey(String mykey) {

    {

        try {

            key = mykey.getBytes(UTF_8);

            sha MessageDigest.getInstance(CIPHER_ALGORITHM_SHA);

            key sha.digest(key);

            key Arrays.copyOf(key16);

            secretKeySpec new SecretKeySpec(keyAES_ALGORITHM);

        catch (NoSuchAlgorithmException e) {

            e.printStackTrace();

        catch (UnsupportedEncodingException e) {

            e.printStackTrace();

        }

    }

}


The CIPHER_ALGORITHM_SHA is "SHA-1".

The base encoding is "UTF-8".

The ES_ALGORITHM is "AES".


When all is set, you can invoke the encrypt and decrypt with the help of doFinal method.

Base64.encodeBase64String(cipher.doFinal(StringToEncrypt.getBytes(UTF_8)));

String(cipher.doFinal(Base64.decodeBase64(encodedText)));


Let me know if anyone needs this utility. I will upload it to my GitHub repo.

Tuesday, 8 June 2021

June 08, 2021

[Resolved] javax.mail.MessagingException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

 If you are using javax mail api for sending, reading mail from your java code and you get stuck with this error while as per your knowledge, all the configuration information is correct, then it might be the issue is with your (corporate) firewall proxy.







Error description: 

javax.mail.MessagingException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate);

  nested exception is:

javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:670)

at javax.mail.Service.connect(Service.java:295)

at javax.mail.Service.connect(Service.java:176)

at ss.EmailUtility.<init>(EmailUtility.java:79)

at email.main(email.java:8)

Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)


That you can resolve with the help of your local network admin.

Meanwhile, to ensure that your code is working fine, what you can do is to bring your code into local machine, where there is no firewall proxy is enabled.

Run your code, and you will see, that there is nothing wrong in your code, it's just the admin setting in your machine which is preventing you to execute the code.