H2 database

To perform integration testing we will be using in memory database H2. First add dependency of h2 in pom.xml

<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>test</scope>
    <version>2.2.220</version>
</dependency>

H2 configurations for integration testing. Add this to application.yml to application.properties under src/test/resources folder. Here we notify Spring to use h2 database with postgresql mode for running integration tests

spring:
  jpa:
    generate-ddl: false
    hibernate.ddl-auto: none    
  sql.init.platform: h2
  datasource:
    driverClassName: org.h2.Driver
    url: jdbc:h2:mem:;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
    username: SA
    password:

Liquibase

Add Liquibase to the configuration. Here we provide the liquibase xml file path which will be read by Spring on running integration tests. All the liquibase migrations will go under the src/test/resources/change-log folder. Create folder change-log under src/test/resources if it doesn’t exist.

spring:
  liquibase:
    enabled: true
    change-log: change-log/changelog-main.xml
    drop-first: true

If the primary key of entity is of type java.util.UUID, uuid-ossp extension has to be added to the postgresql

CREATE EXTENSION IF NOT EXISTS "uuid-ossp" with schema public;

but this extension doesn’t exist in h2 database. So that we don’t encounter this exception on running integration tests we have to create an alias which the h2 database can refer. Create new file uuid-v4-function.xml with alias details for uuid under src/test/resources/change-log folder

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
                   http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <changeSet id="uuid-v4-function" author="test">
        <sql>
            <![CDATA[
            CREATE ALIAS IF NOT EXISTS UUID_GENERATE_V4 FOR "org.h2.value.ValueUuid.getNewRandom";
            ]]>
        </sql>
    </changeSet>
</databaseChangeLog>

Create sequences for the state and country table. Create new file sequences.xml under src/test/resources/change-log/

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <changeSet id="createSequences" author="test">        
        <createSequence sequenceName="country_country_id_seq" startValue="10" incrementBy="50"/>        
        <createSequence sequenceName="state_state_id_seq" startValue="10" incrementBy="50"/>        
    </changeSet>
</databaseChangeLog>

Let’s add test data to state and country tables. Create new file test-data.xml under src/test/resources/change-log/

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <changeSet id="test-data" author="test">
        <sql>            
            insert into country(country_id, created, updated, name)  values (1, current_date, current_date, 'USA');
            insert into country(country_id, created, updated, name) values (2, current_date, current_date, 'India');

            insert into state(state_id, abbreviation, code, name, country_id) values (1, 'TX', 'TX', 'Texas', 1);
            insert into state(state_id, abbreviation, code, name, country_id) values (2, 'AK', 'AK', 'Arkansas', 1);
            insert into state(state_id, abbreviation, code, name, country_id) values (3, 'CA', 'CA', 'California', 1);
            insert into state(state_id, abbreviation, code, name, country_id) values (4, 'PA', 'PA', 'Punjab', 2);
            insert into state(state_id, abbreviation, code, name, country_id) values (5, 'HR', 'HR', 'Haryana', 2);
        </sql>
    </changeSet>
</databaseChangeLog>

Add existing migrations to the changelog-main.xml (create file if it doesn’t exist) file. Here we will refer the main file under the src/main/resources/changelog/main.xml which was created in the previous article [Liquibase for Postgresql with Spring Boot] (/technology/liquibase-for-postgresql-with-springboot.html) which contains all the migrations, so we don’t have to rewrite migrations for tests.

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
                   http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <include file="change-log/uuid-v4-function.xml"/>
    <include file="change-log/sequences.xml"/>
    <include file="changelog/main.xml"/>
    <include file="change-log/test-data.xml"/>    
</databaseChangeLog>

With all these configuration, on running integration test Spring will run changelog-main.xml. Here is the flow of migrations

create alias -> create sequences -> apply all existing migrations -> add test data

Integration test

We will be testing the StateService, if it doesn’t exist create the following two files.

src/main/java/com/demo/service/StateService.java

public interface StateService {    
    List<StateDTO> getStatesByCountry(String name);
}

src/main/java/com/demo/service/impl/StateServiceImp.java

@Service
@Transactional
public class StateServiceImpl implements StateService {

    private final StateRepository stateRepository;

    public StateServiceImpl(StateRepository stateRepository) {
        this.stateRepository = stateRepository;
    }
    
    @Override
    public List<StateDTO> getStatesByCountry(String name) {
        List<State> states = this.stateRepository.findStatesByCountryIsoCode2(name);
        return states.stream().map(state -> new StateDTO(state.getStateId(), state.getName())).toList();
    }
}

Now we have all the configuration wired, we are ready to write the integration test

Create file StateServiceIT.java under src/test/java/com/demo

@RunWith(SpringRunner.class)
@SpringBootTest
public class StateServiceIT {

    @Autowired
    private StateService stateService;

    @Test
    public void getStatesByName() {
        List<StateDTO> stateDTO = stateService.getStatesByCountry("US");
        assertEquals(stateDTO.size(), 3);
    }
}

Size is 3 as we have added 3 states in the test-data.xml. So, we don’t need to write code to push data for testing, as it’s much easier using queries mentioned in the test-data.xml

Conclusion

Here we saw how to write integration tests with liquibase migrations. How it’s easier to set up the test data using the sql queries.

For complete working solution please refer this GitHub Repo

NOTE: This post assumes that you have basic spring boot (v3.0) project setup with all the required dependencies and POSTGRESQl.

Liquibase Configuration for Spring boot

Add following dependency to pom.xml

    <dependency>
        <groupId>org.liquibase</groupId>
        <artifactId>liquibase-core</artifactId>
        <version>4.23.2</version>
    </dependency>

Configure liquibase properties in application.yml or application.properties file

 spring:
   liquibase:
       change-log: classpath:changelog/main.xml
       enabled: true

change-log tells the spring that all the database migrations are in the src/main/resources/changelog/main.xml file

enabled flag if true will run the liquibase configuration when the spring starts

main.xml will contain all the database migrations.

Database Migration

Create file with name ‘1690137730-init-schema.xml’ under src/main/resources/changelog folder and include it in main.xml.
You can name however you want, but it’s good to append timestamp to the file name so it easier to track the migrations.

<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
                   http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <include file="changelog/1690137730-init-schema.xml"/>    
</databaseChangeLog>

Below are the details of 1690137730-init-schema.xml. Here we create two tables country and state and then add foreign key of country in the state table.

Few things to consider

• add unique changeSet id ( usually name of the file should suffice)
• add name of the author
• multiple changesets can be added into single file

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">

    <changeSet id="1690137730-init-sequences" author="my-restaurant" dbms="postgresql">
        <createSequence sequenceName="country_country_id_seq" startValue="1" />
        <createSequence sequenceName="state_state_id_seq" startValue="1" />
    </changeSet>
    <changeSet id="1690137730-init-schema" author="dev">
        <createTable tableName="country">
            <column name="country_id" type="bigint" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="name" type="varchar2(100)">
                <constraints nullable="false"/>
            </column>            
            <column name="updated" type="timestamp" defaultValueComputed="CURRENT_TIMESTAMP"/>
            <column name="created" type="timestamp" defaultValueComputed="CURRENT_TIMESTAMP"/>
        </createTable>

        <createTable tableName="state">
            <column name="state_id" type="bigint" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="name" type="varchar2(100)">
                <constraints nullable="false"/>
            </column>
            <column name="code" type="varchar2(100)"/>            
            <column name="country_id" type="bigint">
                <constraints nullable="false"/>
            </column>
            <column name="updated" type="timestamp" defaultValueComputed="CURRENT_TIMESTAMP"/>
            <column name="created" type="timestamp" defaultValueComputed="CURRENT_TIMESTAMP"/>
        </createTable>
        <addForeignKeyConstraint baseTableName="state" baseColumnNames="country_id" constraintName="fK_state_country_id"
                                 referencedTableName="country"
                                 referencedColumnNames="country_id"/>        
    </changeSet>    
</databaseChangeLog>

Apply Migrations

Assuming that you have valid postgresql connection string in the spring.datasource.url.

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/demodb?currentSchema=myapp

Run the application, this will run the migrations for the above datasource.url connection string. After the application runs successfully sequences and tables should be created under demodb/myapp schema.

Liquibase also creates two extra tables databasechangelog and databasechangeloglock to keep track which migrations have been applied, so that on next turn it won’t apply the migrations again which have been applied.

Entity Reference

Create State and Country entities. These entities will be used by the repositories for spring JPA

@Table(name = "state")
@Entity
@Getter
@Setter
public class State {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "stateSeq")
    @SequenceGenerator(name = "stateSeq", sequenceName = "state_state_id_seq", allocationSize = 1)
    @Column(name = "state_id")
    private Long stateId;

    @Column(name = "name", nullable = false)
    @NotNull
    private String name;

    @Column(name = "code")
    private String code;    

    @ManyToOne
    @JoinColumn(name = "country_id")
    @NotNull
    private Country country;
    
    @Column(name = "created", updatable = false, columnDefinition="TIMESTAMP DEFAULT CURRENT_TIMESTAMP")
    @CreatedDate
    private LocalDateTime created;

    @Column(name = "updated", columnDefinition="TIMESTAMP DEFAULT CURRENT_TIMESTAMP")
    @LastModifiedDate
    private LocalDateTime updated;

}

@Entity
@Table(name = "country")
@Getter
@Setter
public class Country {

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "countrySeq")
    @SequenceGenerator(name = "countrySeq", sequenceName = "country_country_id_seq", allocationSize = 1)
    @Column(name = "country_id")
    private Long countryId;
    
    @Column(name = "name", nullable = false)
    @NotNull
    private String name;
        
    @OneToMany(mappedBy = "country")
    private List<State> states = new ArrayList<>();
    
    @Column(name = "created", updatable = false, columnDefinition="TIMESTAMP DEFAULT CURRENT_TIMESTAMP")
    @CreatedDate
    private LocalDateTime created;

    @Column(name = "updated", columnDefinition="TIMESTAMP DEFAULT CURRENT_TIMESTAMP")
    @LastModifiedDate
    private LocalDateTime updated;
}

Update Migrations

To add another migration, create new file under the changelog folder and include that in the main.xml. Add your changes with new changeset and run the application. On successful start of application new migrations should be applied too.

Conclusion

In this post we saw how keep track of the database migrations using the liquibase tool and integration with spring boot.

AWS S3

1. Create S3 bucket with your domain name. e.g. If your domain is mypersonaldomain.com, then create bucket in S3 with name mypersonaldomain.com

   

2. Allow all public access for this bucket and select the acknowledgement. Hit Create bucket

   

3. Click on the bucket created and go to Properties, at the bottom of the page edit the static website hosting. Enable the static website hosting and enter index.html in the index document and 404.html in the error document. Hit Save Changes. This will create a website endpoint.

   

4. Click the permission tabs, and edit the bucket policy. Add the following policy

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "PublicReadGetObject",
                "Effect": "Allow",
                "Principal": "*",
                "Action": [
                    "s3:GetObject",
                    "s3:PutObjectAcl",
                    "s3:GetObjectAcl"
                ],
                "Resource": "arn:aws:s3:::mypersonaldomain.com/*"
            }
        ]
    }
               
   

   Now our bucket can be accessed publicly and data can be pushed from GitHub action

AWS IAM User

1. Create an IAM user with S3 and Cloudfront permissions. This user will be used by the GitHub actions to copy content from repository to S3 bucket.

2. Create an access key for this user. Note down the access key and access secret (it won’t be generated again)

   

Domain Certificate

We will create certificate for our domain which will used by the cloud distribution

1. Request public certificate in AWS certificate manager.

   

2. Choose DNS verification. ( You can choose any verification)

3. Click on the certificate and copy the CNAME name and values and paste in your Domain DNS to verify.

   

4. Once verified will use this certificate in the cloud distribution

AWS Cloudfront

Now we will create cloud front for our static S3.

1. Create distribution, choose your S3 in the origin domain.
2. Choose your SSL certificate which was created in the previous step
3. Hit Create distribution.
4. Add the distribution domain name in the DNS
   1. Add Alias with value of distribution domain name
   2. Add CNAME for www with value of distribution domain name
5. Edit the cloud distribution, and add alternate domains

   mypersonaldomian.com
   www.mypersonaldomain.com
   

6. After successful save, your domain is now connected to this distribution which is connected to the S3 bucket.

GitHub Action

1. Create a repository and push the code created in previous article to GitHub.
2. Copy values for access id, access secret, bucket name and cloud distribution id

3. Create secrets for the repository with following names

   

4. Create a new GitHub action which will build the code and push the code to AWS S3.

         
    name: CI / CD
       
    # Controls when the action will run. 
    on:
      # Triggers the workflow on push for the main branch
      push:
        branches: [ main ]
       
      # Allows you to run this workflow manually from the Actions tab
      workflow_dispatch:
         
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: 'us-west-2'   
       
    # A workflow run is made up of one or more jobs that can run sequentially or in parallel
    jobs:
      build:
        runs-on: ubuntu-latest
        steps:
        - uses: actions/checkout@v4
        - name: Set up Ruby
          uses: ruby/setup-ruby@v1
          with:
            ruby-version: "3.2.2" # Not needed with a .ruby-version file
            bundler-cache: true
        - name: "Build Site"
          run: bundle exec jekyll build
          env:
            JEKYLL_ENV: production
        - name: "Deploy to AWS S3"
          run: aws s3 sync ./_site/ s3://${{ secrets.AWS_S3_BUCKET_NAME }} --acl public-read --delete --cache-control max-age=604800
        - name: "Invalidate CloudFront Cache"
          run: aws cloudfront create-invalidation --distribution-id ${{ secrets.AWS_CLOUDFRONT_DISTRIBUTION_ID }} --paths "/*"
       
   

5. Run the action, files should be copied to the S3 bucket and the domain should now serve the blog.
6. Any new data pushed to the main branch should automatically build the solution and push the latest to our S3.

Conclusion

In this article we created S3 bucket with our domain name and cloud distribution using amazon managed certificate. And then created the GitHub action which will copy data to our S3 bucket with any new pushed to main branch.

1. First step is to set up the environment with ruby (2.5.0 or higher). Depending on your OS there are couple of ways to install ruby on your local machine. For this article we will be focusing on windows. For different operations systems refer to this Ruby Installation
2. Install RubyGems, GCC and Make

3. Once the ruby in installed next step is to install jekyll

   gem install bundler jekyll

4. Once jekyll is installed, we will use the jekyll cli to create our blog site.

   jekyll new my-blog   
   

5. This will create the folder my-blog in the current working directory. In the my-blog folder you will see autogenerated directories and files which is the skeleton of the jekyll site.

6. Next step is to build all the dependencies for jekyll which can be found in the Gemfile. Run the terminal in admin mode.

   cd my-blog
   bundle install
   bundle exec jekyll serve
   

   This will start serving the files are http://localhost:4000. This will create _site folder which will contain all the compiled files to serve

Minima theme update

As of writing Jekyll uses minima theme (v2.5.1). We will use the latest version of minima theme. As it provides pagination, dark theme, disqus, google analytics and couple of other changes. As the latest version has not been released, so we will fetch the latest from GitHub. Update the minima dependency in Gemfile with following

gem "minima", git: "https://github.com/jekyll/minima"

Understanding the structure

1. _post folder contains all the posts. All the posts should follow the naming convention. YYYY-MM-DD-.markdown. e.g. 2023-03-14-welcome-to-jekyll.markdown.

2. _site folder contains all the compiled pages. By default all the posts are categorized via category/YYYY/MM/DD/name_of_file. This is also the URL for the post. Auto generated post under the _posts folder have 2 categories i.e. jekyll and update. On expanding the _site folder we can see the path of our post

   

This path can be configured using _config.yml file.

Configuration

All the configurations are handled by _config.yml file. Remove twitter_username and github_username as they are deprecated in the latest minima theme.

1. Enable dark mode and social links

      minima:
        skin: auto
        date_format: "%b %-d, %Y"
        social_links:
          - { platform: github,  user_url: "<github_profile_page>" }
          - { platform: twitter, user_url: "<x_profile_page>" }
   

2. Add author details

      author:
        name: Your Name
        email: youremail@email.com
   

Create Posts

1. Create new markdown file under _posts folder and add front matter to top of the file with following name 2023-10-14-my-first-post.markdown.

      ---
      layout: post
      title:  "How to build blog with Jekyll"
      date:   2023-10-14 13:31:06 -0500      
      summary: How to build blog with Jekyll
      categories: jekyll blog
      ---
   

2. Below the front matter write your post.
3. New post will be created under jekyll/blog/2023/10/14/my-first-post.html under the _sites folder.
4. Hit refresh and new post will show up on localhost:4000.

Draft Posts

To create a draft post, create _drafts folder at root level. Create a post under _drafts folder. Draft posts won’t display unless we specify to serve drafts posts. To serve the drafts append –drafts with the serve or build command

bundle exec jekyll serve --drafts

This command will serve drafts and now the draft post will be displayed on the home page.

Conclusion

We saw how to create blog site using jekyll and how to use configuration for dark mode and create new posts and draft posts. In next post we will see how to host the jekyll site to aws using GitHub action How to deploy Jekyll to AWS with Github actions

In the previous blog (POSTGRESQL Full Text Search with Spring Boot - Part 1) we saw the basic setup of full text search in postgresql. In this we will integrate the full text with spring boot using spring jpa.

Prerequisite

• Postgresql
• Spring Boot

Integration

1. Create Spring boot project. There are couple ways to initialize the spring boot.
   1. Spring initializer https://start.spring.io/
   2. Create a spring project from IDE ( IntelliJ or Eclipse).
2. Add Spring JPA dependency

   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-jpa</artifactId>
   </dependency>   
   

3. Create repository for Product.

   @Repository
   public interface ProductRepository extends JpaRepository<Product, UUID> {    
       @Query(value = "select * from product where to_tsvector('english', name) @@ to_tsquery('english', :query) " +
               "or to_tsvector('english', description) @@ to_tsquery('english', :query)", nativeQuery = true)
       List<Product> searchProduct(String query);
   }
   

4. SearchProduct method will return the result using the indexes created in the previous part.

Conclusion

Assuming that our connections are set up with Postgresql, this way we can query the full text indexes in postgresql from spring boot

As the data in the database table starts to grow into millions of rows LIKE query doesn’t perform well. As it can take more than few seconds to minutes in some cases ( depending on the volumn of data in the table) for a single query. To solve this issue it’s best to index data in the column using full text search. Full text search enables faster retreival for search keyword. Most of the databases (eg. MySQL, Postgresql, sql server) come loaded with full text search feature. In this post we will be focusing on Postgresql full text search.

Prerequisite

• Postgresql
• Spring Boot

Full Text search setup

1. Create a table PRODUCT with columns product_id, name and description.

    create table product
    (
        product_id      uuid      default uuid_generate_v4() not null
            primary key,
        name            varchar(255)                         not null,
        description     varchar(1000)                        not null
    )
   

   Run the above query, this will create a table with name of PRODUCT.

   NOTE - if uuid gives error you might have to install the following extension

        CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
   

2. Add data to column

    insert into product(description, name)
    VALUEs ('description 1','Butter Chicken');
       
    insert into product(description, name)
    VALUEs ('description 2','Samosa');
       
    insert into product(description, name)
    VALUEs ('description 3', 'Mango Lassi');
       
    insert into product(description, name)
    VALUEs ('description 4','Mutton Curry');
       
    insert into product(description, name)
    VALUEs ('description 5', 'Tandoori murg');
   

3. add full text index to name and description columns

        CREATE INDEX idx_products_name ON product USING GIN (to_tsvector('english', name));
        CREATE INDEX idx_products_description ON product USING GIN (to_tsvector('english', description));
   

   This will create full text search indexes on the column name and description. English in to_tsvector instructs that the values in the particular column are in English language. So the values in the columns will be tokenized based on the english tokenizer.

4. Query on full text index column

   NOTE: queries are performed on the above 5 row data in step 2

   • The following query will return the result based on the search input

      select * from product where to_tsvector('english', name) @@ to_tsquery('english', 'mutton');
     

   • Typing substring of the query won’t yield any result. e.g. The following query will return 0 rows

      select * from product where to_tsvector('english', name) @@ to_tsquery('english', 'mu');
     

   • To make this scenario work append :* at end of the query. The following query should return 2 rows.

      select * from product where to_tsvector('english', name) @@ to_tsquery('english', 'mu:*');
     

   • To search within multiple columns, use the following query. This should return 5 rows

      select * from product where to_tsvector('english', name) @@ to_tsquery('english', 'de:*')
                or to_tsvector('english', description) @@ to_tsquery('english', 'de:*');
     

Conclusion

In this we created product table and added full text search to name and description columns and also fetched data using the full text search syntax. In the next article we will see how to integrate this into spring boot using spring jpa. POSTGRESQL Full Text Search with Spring Boot - Part 2