Elasticsearch Setup

To setup secure Elasticsearch local setup with Kibana on Windows with powershell.

0. Create Yaml file use with the certutil --in flag named as instances.yml

instances:
  - name: Apollo
    dns: ['localhost']
  - name: devws-kibana
    dns: ['localhost']

1.    Declare variables for use in powershell command

$root = "D:\Tools\E\elasticsearch"
[Version]$esVersion = "6.2.2"
$es = "$root\elasticsearch-$($esVersion.ToString())"
$esService = "elasticsearch_$($esVersion.ToString() -replace '\.','')"
[Version]$KibanaVersion = "6.2.2"
$kibana = "$root\kibana-$($KibanaVersion.ToString())-windows-x86_64"
$kibanaService = "elasticsearch-kibana$($KibanaVersion.ToString() -replace '\.','')"

2.    Install x-pack in elasticsearch

&"$es\bin\elasticsearch-plugin.bat" install x-pack --batch

3.    Create a Self Signed CA certificate.

&"$es\bin\x-pack\certutil.bat" ca --silent --pass password --ca-dn "CN=Elasticsearch-DevWS" --pem --out "$root\elastic-stack-ca.zip"
Expand-Archive -Path "$root\elastic-stack-ca.zip" -DestinationPath "$root\certs"

4.    Create a cert for elasticsearch and kibana

&"$es\bin\x-pack\certutil.bat" cert --silent --pem --ca-cert "$root\certs\ca\ca.crt" --ca-key "$root\certs\ca\ca.key" -in "$root\instances.yml" --ca-pass password --pass password --out "$root\certificate-bundle.zip"
Expand-Archive -Path "$root\certificate-bundle.zip" -DestinationPath "$root\certs"

5.    Copy Certs to proper directories

mkdir "$es\config\certs"
Copy-Item -Path "$root\certs\ca\ca.crt" -Destination "$es\config\certs\ca.crt"
Copy-Item -Path "$root\certs\Apollo\*" -Destination "$es\config\certs\"
mkdir "$kibana\config\certs"
Copy-Item -Path "$root\certs\ca\ca.crt" -Destination "$kibana\config\certs\ca.crt"
Copy-Item -Path "$root\certs\devws-kibana\*" -Destination "$kibana\config\certs\"

6.    Update Elasticsearch.yml to below

cluster.name: DiiConsentes
node.name: Apollo
network.host: localhost
http.port: 9210
discovery.zen.ping.unicast.hosts: [ 'localhost' ]
processors: 2
node.master: true
node.data: true
node.max_local_storage_nodes: 1
xpack.ssl.key: certs/Apollo.key
xpack.ssl.certificate: certs/Apollo.crt
xpack.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.enabled: true
xpack.ssl.key_passphrase: password

7.    Add secure key passphrase to keystore

"password" |  &"$es\bin\elasticsearch-keystore.bat" add xpack.ssl.secure_key_passphrase --stdin
&"$es\bin\elasticsearch-keystore.bat" list

8.    Start Elasticsearch

9.    Set password for build in accounts

$url = "https://localhost:9210/"

$output = & cmd.exe /C "$es\bin\x-pack\setup-passwords.bat auto --url $url -batch" 2>&1

$ Write-Host -ForegroundColor Green -BackgroundColor Black  $output

Output:
12:11:58.636 [main] WARN  org.elasticsearch.deprecation.common.settings.Settings - [key_passphrase] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version. Changed password for user kibana PASSWORD kibana = U6QvLg7w4Cy3pNCCMCMm  Changed password for user logstash_system PASSWORD logstash_system = LPBjX3huDyhPKFFAjuP2 Changed password for user elastic PASSWORD elastic = K7gET7kaBDWw73tm8PEA

 

10.    Parse passwords from response and save to temp files.

$elasticPassword = ($output | Select-String -Pattern "^PASSWORD\selastic\s=\s(.*)$" -AllMatches).Matches[0].Groups[1].Value
$kibanaPassword = ($output | Select-String -Pattern "^PASSWORD\skibana\s=\s(.*)$" -AllMatches).Matches[0].Groups[1].Value
$elasticPassword | Out-File -FilePath "$es\config\elastic.password" -Encoding utf8
$kibanaPassword | Out-File -FilePath "$kibana\config\kibana.password" -Encoding utf8

11.    Remove setting xpack.ssl.key_passphrase from Elasticsearch.yml

12.    Restart Elasticsearch

13.    Verify Elasticsearch is work (and it is)

14.    Install X-Pack in kibana

&"$kibana\bin\kibana-plugin.bat" install x-pack

15.    Update Kibana.yml to below

server.name: devws-kibana
server.host: localhost
elasticsearch.url: https://localhost:9210/
elasticsearch.username: kibana
elasticsearch.password: K7gET7kaBDWw73tm8PEA
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt

16.    Start Kibana

17.    Verify Kibana is running and I am able to log in with elastic user

18.    Stop Kibana

19.    Update Kibana.yml to below [update all path to absolute one]

server.name: devws-kibana
server.host: localhost
server.ssl.enabled: true
server.ssl.certificate: ../config/certs/devws-kibana.crt
server.ssl.key: ../config/certs/devws-kibana.key
elasticsearch.url: https://localhost:9210/
elasticsearch.username: kibana
elasticsearch.password: K7gET7kaBDWw73tm8PEA
elasticsearch.ssl.certificateAuthorities: ../config/certs/ca.crt
xpack.security.encryptionKey: 3qrb1xee9ue9rrh3p93ykj28otgp676iu0l8ziifjopfov6h4sv9jhyp49gpm90t

20.    Generate jks file

keytool -import -v -trustcacerts -alias server-alias -file "$root\certs\ca.crt" -keystore "$root\certs\cacerts.jks" -keypass changeit -storepass changeit

Reference: https://discuss.elastic.co/t/kibana-ssl-pem-error-on-windows/117851

Questions & Answers

Java Q&A

1. equals() implemented and hashCode() not, Is there any issue, if no collection in use?
2. Difference b/w JAX-WS ans JAX-RS
3. Difference b/w  load() and get()
4. How to test database by unittest.
5. Which RunTimeException may be thrown by java.lang.wait() method (java.lang.IllegalMonitorStateException)
6.  Exceptions in finalize method are ignored.
7. what is Dynamic binding?
   Dynamic Binding refers to the case where compiler is not able to resolve the call and the binding is done at run-time only. (as overridden method call decided by run-time object)
8. If we have 3 interfaces A,B,C which are having common method - display() and we are implementing these 3 interfaces in interface D then how this will be handled and which display() will be implemented.
• The first interface in order must be implemented
9. How to make a class non extendable?
   
   • Make the class as Final
10. What is difference between ClassNotFoundException and ClassDefNotFoundException?
    
    • ClassNotFoundException: class not found at run-time in class-path using class.forName(...) or loadClass() methods.
    • ClassDefNotFoundException: class not found at run-time in class-path and in present at compile time. class is referenced with Java’s “new” operator (i.e. static loading).
11. What is difference between HashMap, HashTable & ConcurrentHashMap?
• HashTable: maps keys to values. Any non-null object can be used as a key or as a value.
• HashMap: The HashMap class is roughly equivalent to Hashtable, except that it is unsynchronized (Not Thread-safe) and permits nulls
• ConcurrentHashMap: Even though all operations are thread-safe, retrieval operations do not entail locking, and there is not any support for locking the entire table in a way that prevents all access. This class is fully interoperable with Hashtable in programs that rely on its thread safety but not on its synchronization details.

  Conclusion: If a thread-safe implementation is not needed, it is recommended to use HashMap in place of Hashtable. If a thread-safe highly-concurrent implementation is desired, then it is recommended to use ConcurrentHashMap in place of Hashtable.

12. What is Error ?
1. Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch.
13. How hashCode is calculated for custom object like Employee.

Install java in Linux

Download jdk_version.tar.gz from oracle and extract it to the /user/local/java and create the symlink

user@group:/user/local/java$ tar zxvf /path/to/jdk_version.tar.gz
user@group:/user/local/java$ ln -s jdk_version jdk

user@group:/user/local/java$ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk/bin/java" 1
user@group:/user/local/java$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk/bin/javac" 1
user@group:/user/local/java$ sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk/bin/javaws" 1
user@group:/user/local/java$ sudo update-alternatives --set java /usr/local/java/jdk/bin/java
user@group:/user/local/java$ sudo update-alternatives --set javac /usr/local/java/jdk/bin/javac
user@group:/user/local/java$ sudo update-alternatives --set javaws /usr/local/java/jdk/bin/javaws

Reference# http://www.wikihow.com/Install-Oracle-Java-on-Ubuntu-Linux

Linux Commands

Completely remove an application

• dpkg --purge --force-depends 
• sudo apt-get purge --auto-remove 
• sudo apt-get remove 
• sudo apt-get remove --purge 
• dpkg --remove --force-remove-reinstreq 

remove-reinstreq: Remove a package, even if it's broken

Open Source Softwares (OSS)

Open Source Softwares (OSS)

• can be user free of charge
• is subject to license terms
• may be modifed and passed to anyone
• Is type of software whose source is made freely available
• There are upto 200 different license(divided in to 5 groups) in existence.

1. Strong Copyleft: ex. GPL, CPL [Licenses with a strong copyleft clause stipulate that all modified versions (where these are distributed and made available to the general public) must be subject to the original license.]
2. Restricted Copyleft: LGPL (Lesser GPL), MPL(mozill public lic) [where the original license must be imposed on all modified versions of the software if they are distributed]
3. No Copyleft: BSD, Apache [do not carry any obligation to make the newly added or modified code likewise subject to an OSS license.]
4. OSS with options: Perl Artistic, Clarified Artistic [This group is a "catch-all term" for any licenses that cannot be assigned to any other group.]
5. OSS with Privileges: e.g., software companies such as Netscape. [whose exploitation rights are reserved by the authors. e.g., by allowing them to use versions modified by external programmers as if they were their own property.]

• GPL v2 : OSS licenses within this group contain different license obligations.
• If no derivative work is created, the original GPL component remains subject to the GPL, but the modified proprietary component can be distributed under any license (including a proprietary one).
• Different licenses cannot be combined if they contain conflicting license terms.
• OSS officer has to scan the software purchasing/sales/distribution. (OSS Management)

Additional Info

• Copyleft "Copyleft" means that a piece of software that is subject to certain OSS license conditions can only be distributed under the same or compatible OSS license terms as the case may be.

• Derivative work The term "derivative work" or "derivative" refers to software based on an OSS and constituting an extension, development, modification or other processing thereof.

PermGen setup

Memory Setup

Jvm

> java -Xmx2g -Xms512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Maven

Windows

> set MAVEN_OPTS=-Xmx2g -Xms512M -XX:MaxPermSize=3g -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError

Linux

> export MAVEN_OPTS=-Xmx2g -Xms512M -XX:MaxPermSize=3g -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError

 Tomcat

Windows

> set JAVA_OPTS=-Xmx2g -Xms512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Linux

> export JAVA_OPTS=-Xmx2g -Xms512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

Kill Windows Service

Kill Windows Service

Open command prompt in windows. Windows > Run > cmd > [Enter]

> sc queryex
SERVICE_NAME:
        TYPE               : 110  WIN32_OWN_PROCESS  (interactive)        STATE              : 4  RUNNING                                (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)        WIN32_EXIT_CODE    : 0  (0x0)        SERVICE_EXIT_CODE  : 0  (0x0)        CHECKPOINT         : 0x0        WAIT_HINT          : 0x0        PID                :
        FLAGS              :
> taskkill /f /pid 

Or If you know the image name

> taskkill /f /im <image_name>