Elasticsearch Setup

To setup secure Elasticsearch local setup with Kibana on Windows with powershell.

0. Create Yaml file use with the certutil --in flag named as instances.yml

instances:
  - name: Apollo
    dns: ['localhost']
  - name: devws-kibana
    dns: ['localhost']

1.    Declare variables for use in powershell command

$root = "D:\Tools\E\elasticsearch"
[Version]$esVersion = "6.2.2"
$es = "$root\elasticsearch-$($esVersion.ToString())"
$esService = "elasticsearch_$($esVersion.ToString() -replace '\.','')"
[Version]$KibanaVersion = "6.2.2"
$kibana = "$root\kibana-$($KibanaVersion.ToString())-windows-x86_64"
$kibanaService = "elasticsearch-kibana$($KibanaVersion.ToString() -replace '\.','')"

2.    Install x-pack in elasticsearch

&"$es\bin\elasticsearch-plugin.bat" install x-pack --batch

3.    Create a Self Signed CA certificate.

&"$es\bin\x-pack\certutil.bat" ca --silent --pass password --ca-dn "CN=Elasticsearch-DevWS" --pem --out "$root\elastic-stack-ca.zip"
Expand-Archive -Path "$root\elastic-stack-ca.zip" -DestinationPath "$root\certs"

4.    Create a cert for elasticsearch and kibana

&"$es\bin\x-pack\certutil.bat" cert --silent --pem --ca-cert "$root\certs\ca\ca.crt" --ca-key "$root\certs\ca\ca.key" -in "$root\instances.yml" --ca-pass password --pass password --out "$root\certificate-bundle.zip"
Expand-Archive -Path "$root\certificate-bundle.zip" -DestinationPath "$root\certs"

5.    Copy Certs to proper directories

mkdir "$es\config\certs"
Copy-Item -Path "$root\certs\ca\ca.crt" -Destination "$es\config\certs\ca.crt"
Copy-Item -Path "$root\certs\Apollo\*" -Destination "$es\config\certs\"
mkdir "$kibana\config\certs"
Copy-Item -Path "$root\certs\ca\ca.crt" -Destination "$kibana\config\certs\ca.crt"
Copy-Item -Path "$root\certs\devws-kibana\*" -Destination "$kibana\config\certs\"

6.    Update Elasticsearch.yml to below

cluster.name: DiiConsentes
node.name: Apollo
network.host: localhost
http.port: 9210
discovery.zen.ping.unicast.hosts: [ 'localhost' ]
processors: 2
node.master: true
node.data: true
node.max_local_storage_nodes: 1
xpack.ssl.key: certs/Apollo.key
xpack.ssl.certificate: certs/Apollo.crt
xpack.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.enabled: true
xpack.ssl.key_passphrase: password

7.    Add secure key passphrase to keystore

"password" |  &"$es\bin\elasticsearch-keystore.bat" add xpack.ssl.secure_key_passphrase --stdin
&"$es\bin\elasticsearch-keystore.bat" list

8.    Start Elasticsearch

9.    Set password for build in accounts

$url = "https://localhost:9210/"

$output = & cmd.exe /C "$es\bin\x-pack\setup-passwords.bat auto --url $url -batch" 2>&1

$ Write-Host -ForegroundColor Green -BackgroundColor Black  $output

Output:
12:11:58.636 [main] WARN  org.elasticsearch.deprecation.common.settings.Settings - [key_passphrase] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version. Changed password for user kibana PASSWORD kibana = U6QvLg7w4Cy3pNCCMCMm  Changed password for user logstash_system PASSWORD logstash_system = LPBjX3huDyhPKFFAjuP2 Changed password for user elastic PASSWORD elastic = K7gET7kaBDWw73tm8PEA

 

10.    Parse passwords from response and save to temp files.

$elasticPassword = ($output | Select-String -Pattern "^PASSWORD\selastic\s=\s(.*)$" -AllMatches).Matches[0].Groups[1].Value
$kibanaPassword = ($output | Select-String -Pattern "^PASSWORD\skibana\s=\s(.*)$" -AllMatches).Matches[0].Groups[1].Value
$elasticPassword | Out-File -FilePath "$es\config\elastic.password" -Encoding utf8
$kibanaPassword | Out-File -FilePath "$kibana\config\kibana.password" -Encoding utf8

11.    Remove setting xpack.ssl.key_passphrase from Elasticsearch.yml

12.    Restart Elasticsearch

13.    Verify Elasticsearch is work (and it is)

14.    Install X-Pack in kibana

&"$kibana\bin\kibana-plugin.bat" install x-pack

15.    Update Kibana.yml to below

server.name: devws-kibana
server.host: localhost
elasticsearch.url: https://localhost:9210/
elasticsearch.username: kibana
elasticsearch.password: K7gET7kaBDWw73tm8PEA
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt

16.    Start Kibana

17.    Verify Kibana is running and I am able to log in with elastic user

18.    Stop Kibana

19.    Update Kibana.yml to below [update all path to absolute one]

server.name: devws-kibana
server.host: localhost
server.ssl.enabled: true
server.ssl.certificate: ../config/certs/devws-kibana.crt
server.ssl.key: ../config/certs/devws-kibana.key
elasticsearch.url: https://localhost:9210/
elasticsearch.username: kibana
elasticsearch.password: K7gET7kaBDWw73tm8PEA
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt
xpack.security.encryptionKey: 3qrb1xee9ue9rrh3p93ykj28otgp676iu0l8ziifjopfov6h4sv9jhyp49gpm90t

20.    Generate jks file

keytool -import -v -trustcacerts -alias server-alias -file "$root\certs\ca.crt" -keystore "$root\certs\cacerts.jks" -keypass changeit -storepass changeit

Reference: https://discuss.elastic.co/t/kibana-ssl-pem-error-on-windows/117851

Questions & Answers

Java Q&A

1. equals() implemented and hashCode() not, Is there any issue, if no collection in use?
2. Difference b/w JAX-WS ans JAX-RS
3. Difference b/w  load() and get()
4. How to test database by unittest.
5. Which RunTimeException may be thrown by java.lang.wait() method (java.lang.IllegalMonitorStateException)
6.  Exceptions in finalize method are ignored.
7. what is Dynamic binding?
   Dynamic Binding refers to the case where compiler is not able to resolve the call and the binding is done at run-time only. (as overridden method call decided by run-time object)
8. If we have 3 interfaces A,B,C which are having common method - display() and we are implementing these 3 interfaces in interface D then how this will be handled and which display() will be implemented.
• The first interface in order must be implemented
9. How to make a class non extendable?
   
   • Make the class as Final
10. What is difference between ClassNotFoundException and ClassDefNotFoundException?
    
    • ClassNotFoundException: class not found at run-time in class-path using class.forName(...) or loadClass() methods.
    • ClassDefNotFoundException: class not found at run-time in class-path and in present at compile time. class is referenced with Java’s “new” operator (i.e. static loading).
11. What is difference between HashMap, HashTable & ConcurrentHashMap?
• HashTable: maps keys to values. Any non-null object can be used as a key or as a value.
• HashMap: The HashMap class is roughly equivalent to Hashtable, except that it is unsynchronized (Not Thread-safe) and permits nulls
• ConcurrentHashMap: Even though all operations are thread-safe, retrieval operations do not entail locking, and there is not any support for locking the entire table in a way that prevents all access. This class is fully interoperable with Hashtable in programs that rely on its thread safety but not on its synchronization details.

  Conclusion: If a thread-safe implementation is not needed, it is recommended to use HashMap in place of Hashtable. If a thread-safe highly-concurrent implementation is desired, then it is recommended to use ConcurrentHashMap in place of Hashtable.

12. What is Error ?
1. Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch.
13. How hashCode is calculated for custom object like Employee.

Install java in Linux

Download jdk_version.tar.gz from oracle and extract it to the /user/local/java and create the symlink

user@group:/user/local/java$ tar zxvf /path/to/jdk_version.tar.gz
user@group:/user/local/java$ ln -s jdk_version jdk

user@group:/user/local/java$ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk/bin/java" 1
user@group:/user/local/java$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk/bin/javac" 1
user@group:/user/local/java$ sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk/bin/javaws" 1
user@group:/user/local/java$ sudo update-alternatives --set java /usr/local/java/jdk/bin/java
user@group:/user/local/java$ sudo update-alternatives --set javac /usr/local/java/jdk/bin/javac
user@group:/user/local/java$ sudo update-alternatives --set javaws /usr/local/java/jdk/bin/javaws

Reference# http://www.wikihow.com/Install-Oracle-Java-on-Ubuntu-Linux

Linux Commands

Completely remove an application

• dpkg --purge --force-depends 
• sudo apt-get purge --auto-remove 
• sudo apt-get remove 
• sudo apt-get remove --purge 
• dpkg --remove --force-remove-reinstreq 

remove-reinstreq: Remove a package, even if it's broken

Open Source Softwares (OSS)

Open Source Softwares (OSS)

• can be user free of charge
• is subject to license terms
• may be modifed and passed to anyone
• Is type of software whose source is made freely available
• There are upto 200 different license(divided in to 5 groups) in existence.

1. Strong Copyleft: ex. GPL, CPL [Licenses with a strong copyleft clause stipulate that all modified versions (where these are distributed and made available to the general public) must be subject to the original license.]
2. Restricted Copyleft: LGPL (Lesser GPL), MPL(mozill public lic) [where the original license must be imposed on all modified versions of the software if they are distributed]
3. No Copyleft: BSD, Apache [do not carry any obligation to make the newly added or modified code likewise subject to an OSS license.]
4. OSS with options: Perl Artistic, Clarified Artistic [This group is a "catch-all term" for any licenses that cannot be assigned to any other group.]
5. OSS with Privileges: e.g., software companies such as Netscape. [whose exploitation rights are reserved by the authors. e.g., by allowing them to use versions modified by external programmers as if they were their own property.]

• GPL v2 : OSS licenses within this group contain different license obligations.
• If no derivative work is created, the original GPL component remains subject to the GPL, but the modified proprietary component can be distributed under any license (including a proprietary one).
• Different licenses cannot be combined if they contain conflicting license terms.
• OSS officer has to scan the software purchasing/sales/distribution. (OSS Management)

Additional Info

• Copyleft "Copyleft" means that a piece of software that is subject to certain OSS license conditions can only be distributed under the same or compatible OSS license terms as the case may be.

• Derivative work The term "derivative work" or "derivative" refers to software based on an OSS and constituting an extension, development, modification or other processing thereof.

PermGen setup

Memory Setup

Jvm

> java -Xmx2g -Xms512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Maven

Windows

> set MAVEN_OPTS=-Xmx2g -Xms512M -XX:MaxPermSize=3g -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError

Linux

> export MAVEN_OPTS=-Xmx2g -Xms512M -XX:MaxPermSize=3g -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError

 Tomcat

Windows

> set JAVA_OPTS=-Xmx2g -Xms512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Linux

> export JAVA_OPTS=-Xmx2g -Xms512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Kill Windows Service

Kill Windows Service

Open command prompt in windows. Windows > Run > cmd > [Enter]

> sc queryex
SERVICE_NAME:
        TYPE               : 110  WIN32_OWN_PROCESS  (interactive)        STATE              : 4  RUNNING                                (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)        WIN32_EXIT_CODE    : 0  (0x0)        SERVICE_EXIT_CODE  : 0  (0x0)        CHECKPOINT         : 0x0        WAIT_HINT          : 0x0        PID                :
        FLAGS              :
> taskkill /f /pid 

Or If you know the image name

> taskkill /f /im <image_name>

Log Viewer for WAS 6.1

Web-based log viewer for IBM Websphere 6.x

Advantages:

- Very helpful for test & development environments to view the log files on the browser
- Log viewer would scan through all available profiles and display the log files
Available at http://hostname:port/logview/

Screenshots:

1) Welcome page

2) Profile display page: Displays the list of server profiles. Please contact the admin team to know the actual profile used for the particular application

3) Log folder view: Displays the log folders available

4) Log file viewer: Either view the log online or download the log file.

Continuous Integration Tool

Following tools used for CI

Open Source

1. Jenkins
2. CruiseControl

Commercial

1. ThoughtWorks’ Go
2. Urbancode’s Anthill Pro
3. Jetbrains’ Team City
4. Microsoft’s Team Foundation Server

There are not much difference in these tools (including open source tools). Depends on the organization, which one to use.

Mysql / Sonar Data Migration

• Take mysql dump and copy the sql file to the other machine

> mysqldump -u root -p --opt [database name] > [database name].sql

• Drop and create clean database

> mysql -u root -p
> DROP DATABASE [database name];
> CREATE DATABASE [database name];

• Create the data

> mysql -u root -p [database name] < /path/to/[database name].sql

Jenkins SonarQube Error

Issue:

Following error while running Sonar via Jenkins.

INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 1:22.624s
Final Memory: 50M/777M
INFO: ------------------------------------------------------------------------
ERROR: Error during Sonar runner execution
ERROR: Unable to execute Sonar
ERROR: Caused by: The svn blame command [svn blame --xml --non-interactive -x -w src/main/java/path/to/package/ClassName.java] failed: svn: E155036: Please see the 'svn upgrade' command
svn: E155036: The working copy at 'C:\Jenkins_Home\jobs\JobName\workspace\src\main\java\path\to\package'
is too old (format 8) to work with client version '1.8.10 (r1615264)' (expects format 31). You need to upgrade the working copy first.

Solution

• Navigate to Manage Jenkins > Configure System
• Look for a 'Subversion Workspace Version' dropdown ( Default: 1.4)
• Change the version to latest (1.8)

Sonar Analysis via Jenkins

PreRequisite:

• Jenkins and SonarQube must be up
• Check following default links
1. Jenkins : http://localhost:8080/
2. Sonar Qube : http://localhost:9000/

Sonar Plugin Installation for Jenkins

Outside Internal Network

• Navigate to Jenkins > Manage Jenkins > Manage Plugins > Available 
• Search "Sonar Plugin" and install the same.
• Restart Jenkins.

Inside Internal Network

• Navigate to Jenkins > Manage Jenkins > Manage Plugins > Advanced
• Type the proxy details inside "HTTP Proxy Configuration" and follow the steps for "Outside Internal Network".

OR

• Find the latest sonar plugin https://updates.jenkins-ci.org/download/plugins/sonar/.
• Save the downloaded *.hpi/*.jpi file for Sonar Plugin.
• Navigate to Jenkins > Manage Jenkins > Manage Plugins > Advanced > Upload Plugin
• Upload the downloaded *.hpi/*.jpi file for Sonar Plugin into Jenkins.
• Restart Jenkins.

OR

• Save the downloaded *.hpi/*.jpi file into the $JENKINS_HOME/plugins directory and Restart Jenkins.
• Restart Jenkins.

Add SonarQube Server Details

• Navigate to Jenkins > Manage Jenkins > Configure System.
• Find the Keyword "Sonar" and check for Sonar Installation and Click on "Add Sonar > Advanced..."

• Fill the fields

• Name : <Sonar name, whatever you want to>
• Server URL :
• Sonar account login :
• Sonar account password:
• Database URL : (say jdbc:mysql://localhost:3306/sonar)
• Database login :
• Database password :
• Database driver : (say com.mysql.jdbc.Driver)

• Click "Save".
• 
  

Add Sonar-Runner Details

• Navigate to Jenkins > Manage Jenkins > Configure System.
• Find the Keyword "Sonar Runner" and check for "Sonar Runner Installation" and Click on "Add Sonar Runner"

• Fill the fields

• Name : <Sonar Runner name, whatever you want to>
• SONAR_RUNNER_HOME : <Sonar Runner Extracted Directory path>

• Click "Save".

Sonar Analysis via Jenkins

• Navigate to Jenkins > Job (Already created before say SampleJob)
• Click on link "SampleJob"
• In the left navigation Menu, click "Configure".
• Click to "Add build step", a list will open
• Then click "Invoke Standalone Sonar Analysis".
• Add the details to run Sonar.
• Run the Jon (SampleJob).

SonarQube Install

SonarQube (previously known as "Sonar") Install

SonarQube (previously known as "Sonar")

• Download and extract the latest sonarqube.zip http://dist.sonar.codehaus.org/sonarqube-x.x.zip.
• Edit <install_directory>/conf/sonar.properties to configure the database settings.

Say DB available for sonar is mysql then

sonar.jdbc.username=sonaruser

sonar.jdbc.password=sonarpwd

sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar

[Better to uncomment above lines in the default sonar.properties file] 

More advance features (as host, port, context, java options, proxy...) can be edited in this files.

• To start sonarqube on windows 64bit(choose the folder according to your OS).

C:>"<install_directory>\bin\windows-x86-64\StartSonar.bat" 

• Install sonarqube as service.

C:>"<install_directory>\bin\windows-x86-64\InstallNTService.bat"

• Start sonarqube service

C:>"<install_directory>\bin\windows-x86-64\StartNTService.bat"

Troubleshooting

• I service does not start then
• Open windows services (Window+R > Type services + Enter) and go to SonarQube.
• Open properties > Log On > This account > provide the admin user details for the system.

• Open browser and type http://localhost:9000/ (default)

Jenkins Install

Way 1:

• Download the latest version of Jenkins.zip from http://mirrors.jenkins-ci.org/windows/latest
• Extract Jenkins.zip and run setup.exe to install Jenkins as a service on windows.

Way 2:

• Download the latest version of Jenkins.war from http://mirrors.jenkins-ci.org/war/latest/jenkins.war
• Run the command to start jenkins

c: > java -DJENKINS_HOME=<path_to_jenkins_home> -jar jenkins.war --httpPort=<http_port_to_run_jenkins>

MySQL Install

Mysql Installation

• Extract the install archive to the chosen installation location (Say "C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx" ).
• Copy .ini file (say "my-default.ini") from "...\MySQL Server x.x.xx\my-default.ini" to "...\MySQL Server x.x.xx\data\my-default.ini".
• Edit "...\MySQL Server x.x.xx\data\my-default.ini" and type the following lines

basedir =C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx
datadir =C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx\data

• To start the server, enter command:

C:\> "C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx\bin\mysqld" --console

• To start the mysqld server:

C:\> "C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx\bin\mysqld"

• The path to mysqld may vary depending on the install location of MySQL on your system.
• To stop the mysqld server:

C:\> "C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx\bin\mysqladmin" -u root shutdown

• Add/update "Path" variable in "Environment Variables" - "System Variables" - Path with "C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx\bin".
• Stop the server & Install the server as a service using this command:

C:\> "C:\MyLocation\mysql\MySQL\MySQL Server x.x.xx\bin\mysqld" --install

Maven Install

For Windows:

• You need a Java Development Kit (JDK).
• Explode maven. say C:\Program Files\Apache Software Foundation/apache-maven-X.X.X
• Add the following environment variables.
• Set JAVA_HOME=C:\Program Files\Java\jdk-X.X.X
• Set Path=%Path%;%JAVA_HOME%\bin
• Set M2_HOME=C:\Program Files\Apache Software Foundation\apache-maven-X.X.X
• Set M2=%M2_HOME%\bin
• Set Path=%Path%;%M2%
• Run mvn --version

For Linux:

• Download the latest archive from https://maven.apache.org/download.cgi
• Extract & set the path to maven

~$ tar -xzvf apache-maven-x.y.z-bin.tar.gz

~$ sudo cp -R /extracted/path/to/maven/apache-maven-x.y.z /usr/local

~$ cd /usr/local/apache-maven-x.y.z/bin/

~$ sudo ln -s /usr/local/apache-maven-x.y.z/bin/mvn /usr/bin/mvn

HTML encoder

If you need to encode HTML/CSS/JavaScript code.

say: To write on blog

Use : http://www.opinionatedgeek.com/DotNet/Tools/HTMLEncode/Encode.aspx

encode your code and paste it to blog post.

Create Oracle User

--Selct all users
SELECT * FROM ALL_USERS;
SELECT * FROM SYS.USER$;
/
--To change the password for a user
ALTER USER username IDENTIFIED BY new_password;
/
--To Dropping a Database User CASCADE  Objects
DROP USER user2delete CASCADE;
/
--create new user
--This CREATE USER statement would create a new user called user_apn in the Oracle database whose password is user_apn
DROP USER user_apn CASCADE;
/
CREATE USER user_apn IDENTIFIED BY user_apn;
/
GRANT CONNECT, RESOURCE TO user_apn;
/

Oracle Password Expired

Run the following command

> sqlplus system/admin@localhost/xe

SQL*Plus: Release 11.2.0.2.0 Production on Thu Nov 20 16:36:23 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-28001: the password has expired

Changing password for system
New password:
Retype new password:

or development you can disable password policy if no other profile was set (i.e. disable password expiration in default one):

ALTER PROFILE "DEFAULT" LIMIT PASSWORD_VERIFY_FUNCTION NULL;

Get System ( Desktop / Laptop ) Information

Open the command prompt (windows + R and type cmd) on windows.
The basic command in wmic.

Serial Number

> wmic bios get serialnumber

O/P

SerialNumber

XNNXXXX

Product Id

> wmic os get "SerialNumber"

O/P

SerialNumber
NNNNN-NNN-NNNNNNN-NNNNN

More Info

> wmic computersystem get model,name,manufacturer,systemtype

you can match different options and get more inofrmation.