Ever wondered what actually happens inside a Large Language Model (LLM) when you type a prompt? We often think of AI as an unapproachable, mystical black box. However, the foundational mechanics behind systems like ChatGPT boil down to clean, elegant pattern recognition loops that you can write, observe, and control directly on your standard computer. In this walkthrough, we will pull back the curtain and explore how to build a localized next-token prediction engine from scra

In This blog post I will guide you on how can you improve the performance of spark job

how to create spark Data Frame from a string value in java can be done in few steps like below Here are some code in java programming to create data frame with a string column, since we are not doing any operation on string column so it will go as single column only. String input_string = "anydataflow"; List<Row> input_list=new ArrayList<Row>(); input_list.add(RowFactory.create(input_string)); List<org.apache.spark.sql.types.StructField> listOfStructField= new ArrayList<or

processing XML data is challenging which can be solved in spark using udf and xpath Here we are putting some practical code which can be reused in problem statement to solve xml data challenges. 1. UDF to Convert XML to JSON Here showing java udf which takes xml string as input and generate json string in output. public static void toJsonXml(SparkSession spark) { UDF1 getJsonXml = new UDF1<String, String>() { public String call(final String str) throws Exception { return...

Deployment of Apache Cassandra NoSql database on GCP and use cassandra query language on cloud shell. Let's start instead going with definition and explanation content. Here assumption is that you already have google account with credentials or have lap account setup in place. After login you will be redirected to google cloud console where you will services list left hand side and in header bar search option and some other. Here you one will be cloud shell (click on that to

As most of us already know that IOT getting popular day by day and it's giving good opportunity to make business more stable and managable. There are so many organization who are using IOT devices and getting befitted from sensor and their applications. Still there are many organization who need this popular technological implementation to manage, organize their services/product. There are many IOT dashboard and graphical presentation tools available in market. ThingsBoard is

web page not able to load http://localhost/ ionic issue If you are reading this means i assume you already know about IONIC platform and used to convert you app in android/ios application. When we build ionic application it run successfully on pc's browser and don't give any error. but when we run it in emulator/android/ios phone we mostly get below error: net::ERR_CONNECTION_REFUSED web page not able to load If you are not able to see the app in browser then please resolve t

1. ImportError: cannot import name SparkSession 2. java.io.EOFException 3. database not found , table not found. Hello, today we are going to discuss few APACHE SPARK error which we mostly get when we use Scala or Python API. As of now we are writing for only 3, but there are many more so let us kwon in comment section. 1. When we submit our Pyspark application code to run by "spark-submit" command then we get exception: "ImportError: cannot import name SparkSession" Soluti

Pyspark is combination of 2 part: 1. Spark 2. Python Spark is open source distributed processing framework which do in-memory processing which makes it 10 times faster then existing processing framework Hadoop. Spark is written in Scala programming language so to provide development support in python programming they provided API which makes it easier for python developer to write spark code in Python. Let see the architecture, in which we will process CSV files in Pyspark c

While working with DataFrame & DataSet API, difficult to come back on RDD As everybody focused on spark dataframe API or Dataset API, developer are more focused on the solution based on both of the API's. In some case if we need to deal with rdd then it's getting challenging job to solve the use case. I found below usecase from one of my friend who was struggling to solve this, when I try then it took less then 30 minutes to finish this. I thought to put on blog for the other

Bigram Count Program with Sorting data using Comparator code will be shown in this blog with details explanation. As of now we have seen lot's of example of wordcount MapReduce which is mostly used to explain how MapReduce works in hadoop and how it use the hadoop distributed file system. Here we are going to see next level of WordCount program in term of "BIGRAM"(combination of 2 word's) count. What is BIGRAM: Bigram is sequences of two consecutive words. Here we use MapRedu

Hive: A data warehouse tool on top of Hadoop HDFS (hadoop distributed file system) support most of the useful or I should say SQL like data types. There are following Data Type supported by hive: 1. Numeric 2. Strings 3. Date/Timestamp 4. Boolean 5. Complex I am considering that you already know hive tool and have worked at least used all the type which are mentioned in above except COMPLEX data types. Please comment in this blog if you have any question regarding above data

As Apache Spark became the first choice for data processing and analytics because it is scalable, easy to code, support lot of file formats like avro, parquet, orc, text, json... Mainframe mainly work around cobol and store data in cobol file format. So reading cobol file format in spark is again challenge. Thanks to the team who developing "Cobrix", it is available on github which you can build for your spark version and use to handle MainFrame COBOL date efficiently. Docume

Efficient implementation of data pipeline increase performance of Data warehouse and ease to generate quality KPI. As you already know bigdata is very important these days to analyse huge data in less time and with less effort. We are discussing about a simple use case which can be seen around in real life, which is connecting RDBMS with Hadoop/Spark ecosystem and show the analytical dashboard on BI tool. Architecture Diagram This architecture looks simple in first look but

Thingsboard game changer in IOT space, with open source and enterprise version both. It's easy to use open source customized version as well. It's online and offline both. It is Open source tool can connect multiple sources. Backend database are either Cassandra or postgreSql. We used in some use case after customization and used commercial version as well. Let us know if thinking to use this tool in any use case and need any guide or help. We have helped many organization to

Best practice and point to keep in mind while to setting up secure bigdata hadoop cluster. Welcome to #anydataflow. As the internet world growing the need to use bidata processing system also growing very fast. When we talk about data then the first things comes in mind is how to keep it secure & analyse securely at scale. We are listing some blocks to implement secure bigdata hadoop cluster: “1. Authentication 2. Authorization 3. Encryption in motion 4. Encryption at REST 5.

Launch Hadoop cluster in google cloud platform in few minutes with less configuration and installation of tools . As you all aware that setting up Hadoop cluster means hour's of work, and after that also some time it's getting complicated to use the cluster. To solve this problem Google launched a service called "Dataproc" which can launch hadoop cluster in few minute only. “ Dataproc is a fast, easy-to-use, fully managed cloud service for running Apache Spark and Apache Hado